[Building Sakai] Coping with Full GC's

[Branden Visser]

Hi everyone,

We've been experiencing some mid-day Full GC's lately, despite a reboot 
cycle that ensures no server maintains more than a 48hr uptime. For us, 
a full GC is synonymous with a server crash, and I think that is related 
to the size of our heaps (5GB).

Our environment is documented in PROD-44 on JIRA. I'm wondering if 
anyone has any advice to better tune our JVM's to avoid these scenarios? 
I'm hesitant to just throw more JVM's on our app servers, as I think 6 
instances at 5G should be enough to handle our user-base -- though I 
could be wrong there.

Using JMeter and an assortment of BadBoy-generated test cases, I've been 
unable to reproduce a Full GC. I've maxed out the resources of 2 desktop 
machines (JMeter clients) trying to break a single app server that has a 
reduced heap (3G), so I'm unsure how to reliably reproduce the scenario 
in a way that I can mingle with the configuration and test.

I remember some mention on the lists of triggering a memory dump (kill 
-QUIT?) when a Full GC happens to diagnose where the memory was being 
used. However, in my experience, kill -QUIT's haven't been able to run 
during a full GC until the GC is complete -- is anyone able to comment 
or advise on that?

Best,

Branden