[Building Sakai] 1000s of threads and the org.sakaiproject.site.api.SiteService.userSiteCache

[Matthew Buckett]

Ok we've got a thread dump with native frames and all the threads look
to be like this:

----------------- 6209 -----------------
0x00007fd1d4aea2d4 __pthread_cond_wait + 0xc4
0x00007fd1d39e2a6c _ZN15JvmtiRawMonitor11SimpleEnterEP6Thread + 0xbc
0x00007fd1d39e30ad _ZN15JvmtiRawMonitor9raw_enterEP6Thread + 0xcd
0x00007fd1d39c7a14 _ZN8JvmtiEnv15RawMonitorEnterEP15JvmtiRawMonitor + 0xa4
0x00007fd1d2346368 ????????
0x00007fd1d2334801 ????????
0x00007fd1d2335520 ????????
0x00007fd1d39d4b87 _ZN11JvmtiExport17post_thread_startEP10JavaThread + 0x197
0x00007fd1d3c22100 _ZN10JavaThread3runEv + 0x270
0x00007fd1d3aefb62 _ZL10java_startP6Thread + 0xf2

My current thought is that we have some JVM TI agent loaded that is
watching thread creation and that agent is slowing down the startup of
new threads. The ???? frames in the thread dump point to code that
isn't part of the JVM and doesn't have debugging symbols enabled.

There are also a couple of frames that look similar:

----------------- 6230 -----------------
0x00007fd1d4aea2d4 __pthread_cond_wait + 0xc4
0x00007fd1d39e2a6c _ZN15JvmtiRawMonitor11SimpleEnterEP6Thread + 0xbc
0x00007fd1d39e30ad _ZN15JvmtiRawMonitor9raw_enterEP6Thread + 0xcd
0x00007fd1d39c7a14 _ZN8JvmtiEnv15RawMonitorEnterEP15JvmtiRawMonitor + 0xa4
0x00007fd1d2346368 ????????
0x00007fd1d2334617 ????????
0x00007fd1d2335330 ????????
0x00007fd1d39d4e9d _ZN11JvmtiExport15post_thread_endEP10JavaThread + 0x1bd
0x00007fd1d3c219b6 _ZN10JavaThread4exitEbNS_8ExitTypeE + 0xea6
0x00007fd1d3c21d62 _ZN10JavaThread17thread_main_innerEv + 0x32
0x00007fd1d3c220b1 _ZN10JavaThread3runEv + 0x221
0x00007fd1d3aefb62 _ZL10java_startP6Thread + 0xf2

this frame I think is a thread that is terminating and telling the JVM
TI about it's termination.

-- 
  Matthew Buckett, VLE Developer, IT Services, University of Oxford