[Building Sakai] Java 6 seems to be a problem for us
Earle Nietzel
Earle.Nietzel at marist.edu
Wed Oct 14 07:37:10 PDT 2009
Hi just wanted to chime in since we were having some pause issues that we
recently resolved.
We were using the throughput collector and every couple of minutes users
may experience some pausing for like 2 seconds.
After doing some testing I found these jvm options to be very helpful and
pauses are now non issues:
JAVA_OPTS="-server -d64 -Djava.awt.headless=true
-Djava.library.path=/usr/lib -Dsun.net.inetaddr.ttl=60
-Dsun.rmi.dgc.client.gcInterval=3600000
-Dsun.rmi.dgc.server.gcInterval=3600000 -Xms5120m -Xmx5120m
-XX:NewSize=768m -XX:MaxNewSize=768m -XX:PermSize=512m
-XX:MaxPermSize=512m -XX:+UseConcMarkSweepGC -XX:+CMSIncrementalMode
-XX:+CMSIncrementalPacing -XX:+CMSParallelRemarkEnabled -XX:+UseParNewGC
-XX:SurvivorRatio=6 -XX:+PrintGCDetails -XX:+PrintHeapAtGC
-Xloggc:/var/log/tomcat5/ilearn.vgc"
I found in our testing the sizing the eden space with -XX:NewSize=768m
-XX:MaxNewSize=768m was very important.
I noticed you did not have this in your config?
Also look at this link to help you figure this out:
http://java.sun.com/docs/hotspot/gc1.4.2/example.html
Good Luck,
Earle
T: 845-575-3509
sakai-dev-bounces at collab.sakaiproject.org wrote on 10/14/2009 07:58:28 AM:
> From:
>
> Charles Hedrick <hedrick at rutgers.edu>
>
> To:
>
> Aaron Zeckoski <aaronz at vt.edu>
>
> Cc:
>
> sakai-dev at collab.sakaiproject.org
>
> Date:
>
> 10/14/2009 07:58 AM
>
> Subject:
>
> Re: [Building Sakai] Java 6 seems to be a problem for us
>
> Sent by:
>
> sakai-dev-bounces at collab.sakaiproject.org
>
> I'm not going to be able to do any more to answer, since my user
> population has run out of patience with testing this semester. The
> current settings are running well. Yes, I have spent hours watching
> java with jstat -gc and catalina.out, so I do know how the various
> settings behave.
>
> I didn't let Java choose the garbage collector because the
> documentation suggested that it would always choose parallel. I
> don't agree that the priority is throughput. We have plenty of CPU,
> and adding more is cheap. I don't care about pauses of a second or
> two, but longer cause a bad user experience, and pauses of a minute
> are unacceptable. Users think the system is down (actually the load
> balancer thinks the system is down and switches people, thus
> requiring a new login). I saw long pauses with the parallel GC. I
> haven't seen any so far with CMS with this tuning.
>
> Once having chosen that, I then have to specify survivor space, as
> you note. However this seems not to be true in 6 -- in 5 you have to
> say something or the survivor spaces aren't used, but in 6 the sizes
> chosen are reasonable; it's just that it uses only 1 or 2
> generations, which may not be optimal.
>
> Possibly the use of a large space is out of date. When I was doing
> the primary tuning, some of the code handled uploaded files by
> reading them into a byte array. So it needed to allocate an array
> the size of the file being loaded. We want to support 1 GB files.
> Until we used a large new, we'd get problems when someone uploaded a
> large file. I can't find the URL at the moment, but Sun published a
> white paper giving recommendations for 5 based on experience. They
> also recommended large new. Indeed my settings were taken almost
> directly from that paper.
>
> It is certainly possible that old is too large. I've been slowly
> lowering it. But I've watched the usage after full GC's, and haven't
> been comfortable with much lower. I want a reasonable safety margin.
> With these settings, old is about 9 GB. Since the default starting
> point for CMS in Java 5 is 68%, that's really a maximum size of 6
> GB. I've seen the heap after a full be 4 GB, so I don't think a
> smaller heap makes sense.
>
> Java considers 64-bit Solaris systems to be server, so -server isn't
> needed for us.
>
> On Oct 14, 2009, at 5:42:56 AM, Aaron Zeckoski wrote:
>
> JAVA_OPTS=" -d64 -Xmx12000m -Xms12000m -XX:+UseConcMarkSweepGC
> -XX:+UseParNewGC -XX:CMSInitiatingOccupancyFraction=75
-XX:MaxPermSize=512m
> -XX:PermSize=512m -XX:+DisableExplicitGC
>
> For 6 I decided to let it tune as much as possible, so I started with
>
> JAVA_OPTS=" -d64 -Xmx12000m -Xms12000m -XX:+UseConcMarkSweepGC
> -XX:+UseParNewGC -XX:MaxPermSize=512m -XX:PermSize=64m "
>
> If you run with these settings you will effectively have no survivor
> spaces since the default survivor space ratio when using
> UseConcMarkSweepGC is a very high number. When I say allowing the JVM
> to autotune (via ergonomics) I mean something more like this (as a
> starting point):
> JAVA_OPTS="-server -d64 -Xmx12000m -XX:MaxPermSize=512m
-XX:GCTimeRatio=49"
>
> You may want to allocate the complete memory block by setting the Xms
> in production but you will want to know how much you truly need before
> you set a massive size like 12 GB. It takes a long time to garbage
> collect so much memory and if you can run with less you definitely
> will want to. You would also want to play with the ergonomics
> settings.
> http://java.sun.com/javase/technologies/hotspot/gc/gc_tuning_6.html
>
> I like this article:
>
http://kirk.blog-city.com/advice_on_jvm_heap_tuning_dont_touch_that_dial.htm
>
> I would not leave out the -server option, it should be the first one
> according to Sun. I would also not use the Xmn setting and instead
> adjust ratios. Finally, I would use the parallel collector
> (UseParallelGC) and parallel compactor (UseParallelOldGC) rather than
> the low pause collector (UseConcMarkSweepGC) since in web apps the top
> priority is throughput and not pause times. That's where I would start
> anyway and then begin collecting numbers to see what settings are
> working in my environment.
>
> Did you get any charts of the memory spaces over time when you were
> running with the trimmed down settings? Do you have any for the
> current settings (over maybe 24 hours)?
> -AZ
>
>
> On Wed, Oct 14, 2009 at 5:16 AM, Charles Hedrick <hedrick at rutgers.edu>
wrote:
> On Oct 13, 2009, at 11:18:37 AM, Aaron Zeckoski wrote:
>
> Have you tried running without the tuning params and seeing where the
> auto tuning settles? You are currently forcing the JVM to run a series
> of params that might be brutally inefficient and it can be eye-opening
> to see how the JVM tunes itself. You may have already done this and
> this might be how you settled on these numbers but I thought I would
> ask anyway.
>
> With 5 I started with recommendations from a Sun tuning paper. In 5 more
> tuning is needed than in 6. Indeed 6 tends to default to something close
to
> the recommendations we followed for tuning 5.
>
> For 5:
>
> -d64 -Xmx12000m -Xms12000m is necessary.
>
> -XX:+UseConcMarkSweepGC -XX:+UseParNewGC is the basic choice of GCs,
which
> won't default in 5.
>
> -Xmn3g was done fairly early, to give us a large enough eden to survive
> uploading large files. I believe we allow 1 GB uploads. We had problems
> without this.
>
> -XX:SurvivorRatio=8 -XX:TargetSurvivorRatio=90
-XX:MaxTenuringThreshold=15
> is from a Sun tuning paper. In 6 we tried without it, and in
retrospect
> I'm not sure it's needed in 6. However in 6, it tends to default to one
or
> two generations, which a number of people think is too small.
>
> -XX:MaxPermSize=512m -XX:PermSize=512m max is certainly needed. I've
> watched with jstat and seen both the full gc's caused by starting with a
> small permsize, and the pauses caused by those GCs.
>
> -XX:+UseMembar -XX:-UseThreadPriorities is probably unnecesary, as
I've
> said. However I'd tried it both way and not seen any performance
problem.
> I'd probably start without this, particularly in 6.
>
> -XX:+DisableExplicitGC is recommended commonly. We really don't want
audio
> uploads in Samigo to trigger a full GC (this is fixed in trunk).
>
> For 6, if we get the pauses fixed so we can go back I think the
following
> would do:
>
> JAVA_OPTS=" -d64 -Xmx12000m -Xms12000m -XX:+UseConcMarkSweepGC
> -XX:+UseParNewGC -XX:CMSInitiatingOccupancyFraction=75
-XX:MaxPermSize=512m
> -XX:PermSize=512m -XX:+DisableExplicitGC
>
> For 6 I decided to let it tune as much as possible, so I started with
>
> JAVA_OPTS=" -d64 -Xmx12000m -Xms12000m -XX:+UseConcMarkSweepGC
> -XX:+UseParNewGC -XX:MaxPermSize=512m -XX:PermSize=64m "
>
> I can tell you that none of the changes caused a problem. Some are
probably
> just unnecessary. The things I feel strongly about in 6 based on our
> experience are the CMSInitiatingOccupancyFraction and starting permsize
> larger. I think you'll find a number of experts that think the default
> initiating point is way too high in 6, and that too few survivor
generations
> are used by default. At any rate, with the default
> CMSInitiatinngOccupancyFraction we saw slow full GCs. I suspected memory
> fragmentation, but it's possible that whatever problem is resulting in
our
> slow minor GCs was actually at fault, and when that is fixed we won't
need
> this either.
>
>
>
>
> --
> Aaron Zeckoski (azeckoski (at) vt.edu)
> Senior Research Engineer - CARET - University of Cambridge
> https://twitter.com/azeckoski - http://www.linkedin.com/in/azeckoski
> http://aaronz-sakai.blogspot.com/ - http://tinyurl.com/azprofile
> [attachment "smime.p7s" deleted by Earle Nietzel/ADM/Marist]
> _______________________________________________
> sakai-dev mailing list
> sakai-dev at collab.sakaiproject.org
> http://collab.sakaiproject.org/mailman/listinfo/sakai-dev
>
> TO UNSUBSCRIBE: send email to sakai-dev-
> unsubscribe at collab.sakaiproject.org with a subject of "unsubscribe"
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://collab.sakaiproject.org/pipermail/sakai-dev/attachments/20091014/2438999c/attachment.html
More information about the sakai-dev
mailing list