[Building Sakai] JMX performance concerns

[Branden Visser]

For posterity, I'll just add that we've enabled JMX on our production 
cluster and have no more performance concerns after setting the argument 
-XX:+DisableExplicitGC.

Thanks to everyone for your replies.

Best,
Branden

Branden Visser wrote:
> I think I found the culprit. Remote JMX monitoring requires RMI, which 
> is probably triggering this:
> 
> http://bugs.sun.com/bugdatabase/view_bug.do?bug_id=6200091
> 
> Increasing the gcInterval for the explicit RMI collection doesn't seem 
> to have any effect either. I've added -XX:+DisableExplicitGC and it 
> looks to be a winner.
> 
> Assuming that RMI invokes a timed GC for a reason, I wonder what kind of 
> garbage could be left behind from this? Maybe a non-issue given the 
> nature of JMX (i.e., only seldomly connected)?
> 
> Thanks,
> Branden
> 
> Branden Visser wrote:
>> Hi Alan,
>>
>> With Java 1.5.0_21 (most recent 1.5 version), I'm getting the same 
>> thing. I've pasted a snip of stdout during start-up [2].
>>
>> This JVM's tuning [1] works fine in production.
>>
>> Thanks,
>> Branden
>>
>> [1] http://jira.sakaiproject.org/browse/PROD-44
>>
>> [2]:
>>
>> INFO: Starting up MyFaces-package : myfaces-impl in version : 1.1.5 
>> from path : 
>> file:/sfw/sakai/sakai24/qa/clew25/tomcat1/webapps/sakai-calendar-summary-too$ 
>>
>> INFO: MyFaces-package : tomahawk-sandbox not found. (2010-01-22 
>> 08:41:39,606 main_org.apache.myfaces.config.FacesConfigurator)
>> INFO: Starting up MyFaces-package : tomahawk in version : 1.1.6 from 
>> path : 
>> file:/sfw/sakai/sakai24/qa/clew25/tomcat1/webapps/sakai-calendar-summary-tool/WE$ 
>>
>> [Unloading class GregorSamsa$4]
>> [Unloading class GregorSamsa]
>> [Unloading class GregorSamsa$2]
>> [Unloading class GregorSamsa$0]
>> [Unloading class GregorSamsa]
>> [Unloading class GregorSamsa]
>> [Unloading class GregorSamsa$0]
>> [Unloading class GregorSamsa]
>> [Unloading class GregorSamsa$3]
>> [Unloading class GregorSamsa]
>> [Unloading class GregorSamsa$4]
>> [Unloading class GregorSamsa$1]
>> [Unloading class GregorSamsa]
>> [Unloading class GregorSamsa]
>> [Unloading class GregorSamsa]
>> [Unloading class GregorSamsa]
>> [Unloading class GregorSamsa]
>> [Unloading class GregorSamsa]
>> [Unloading class GregorSamsa]
>> [Unloading class GregorSamsa$1]
>> [Unloading class GregorSamsa]
>> [Unloading class GregorSamsa]
>> [Unloading class GregorSamsa$2]
>> [Unloading class GregorSamsa$3]
>> INFO: Serialization provider : class 
>> org.apache.myfaces.shared_impl.util.serial.DefaultSerialFactory]
>>
>> Berg, A.M. wrote:
>>> Hi Branden,
>>>
>>> We use JMX at UvA under fire, but have not seen this. If you have not 
>>> already, can you update your Java version and try again. I am very 
>>> curious to hear what the issue is.
>>>
>>> Alan
>>>
>>> Alan Berg
>>> Interim QA Director - The Sakai Foundation
>>>
>>> Senior Developer / Quality Assurance
>>> Group Education and Research Services
>>> Central Computer Services
>>> University of Amsterdam
>>>
>>> http://home.uva.nl/a.m.berg
>>>
>>>
>>>
>>>
>>> -----Original Message-----
>>> From: sakai-dev-bounces at collab.sakaiproject.org on behalf of Branden 
>>> Visser
>>> Sent: Thu 1/21/2010 17:35
>>> To: Sakai
>>> Subject: [Building Sakai] JMX performance concerns
>>>  
>>> Hi all,
>>>
>>> I've enabled JMX on a test server, and found that it causes a lot of 
>>> class unloading even with no active users:
>>>
>>> [Unloading class sun.reflect.GeneratedConstructorAccessor664]
>>>
>>> and the server even eventually crashes:
>>>
>>> #
>>> # An unexpected error has been detected by HotSpot Virtual Machine:
>>> #
>>> #  SIGSEGV (0xb) at pc=0xffffffff7e7dc8b0, pid=27373, tid=6
>>> #
>>> # Java VM: Java HotSpot(TM) 64-Bit Server VM (1.5.0_13-b05 mixed mode)
>>> # Problematic frame:
>>> # V  [libjvm.so+0x7dc8b0]
>>> #
>>> # An error report file with more information is saved as 
>>> hs_err_pid27373.log
>>>
>>> hs_err_pid27373.log suggests that it was a garbage collection thread 
>>> that crashed the server. I followed the instructions on confluence 
>>> [1] to set up remote JMX profiling.
>>>
>>> It does this if there is nothing connected to the JMX console as well.
>>>
>>> Any ideas?
>>>
>>> Thanks,
>>> Branden
>>>
>>> [1] 
>>> http://confluence.sakaiproject.org/display/QA/Remote+JVM+profiling+via+SSH+tunnels 
>>>
>>> _______________________________________________
>>> sakai-dev mailing list
>>> sakai-dev at collab.sakaiproject.org
>>> http://collab.sakaiproject.org/mailman/listinfo/sakai-dev
>>>
>>> TO UNSUBSCRIBE: send email to 
>>> sakai-dev-unsubscribe at collab.sakaiproject.org with a subject of 
>>> "unsubscribe"
>>>
>> _______________________________________________
>> sakai-dev mailing list
>> sakai-dev at collab.sakaiproject.org
>> http://collab.sakaiproject.org/mailman/listinfo/sakai-dev
>>
>> TO UNSUBSCRIBE: send email to 
>> sakai-dev-unsubscribe at collab.sakaiproject.org with a subject of 
>> "unsubscribe"