[Building Sakai] tomcat 7 threads maxing out , freezing Sakai instance

Tue Mar 25 00:12:49 PDT 2014

Dear all,
Lately our 2.9.2 Sakai  instances periodically grind to a halt atleast once per day. We have found that when this happens, tomcat will have maxed out the thread count. The DB writes fail from tomcat logs, with the following entry in logs:

...
Caused by: org.apache.commons.dbcp.SQLNestedException: Cannot get a connection, pool error Timeout waiting for idle object
        at org.apache.commons.dbcp.PoolingDataSource.getConnection(PoolingDataSource.java:104)
...

Only a restart gets things going.

We tried to increase maxThread counts in tomcat conf, but this did not really help, as eventually they were all used up. We switched to nio connector in tomcat's serve.xml as follows:

 <Connector port="8080"  protocol="org.apache.coyote.http11.Http11NioProtocol"
               connectionTimeout="20000"
               redirectPort="8443" maxThreads="250"/>

Other info:
-----------------
Server version: Apache Tomcat/7.0.41
Server number:  7.0.41.0
OS Name:        SunOS
OS Version:     5.11
Architecture:   x86 (we actually run on 64bit..not sure why it give this)
JVM Version:    1.7.0_21-b11

Any help on this will be appreciated.

This seems to hold on longer, but eventually it still runs out.   I have attached a full thread dump, but here is an excerpt of where things seem to go wrong:

"http-nio-8080-exec-487" daemon prio=3 tid=0x0000000007dc0800 nid=0x25b4 in Object.wait() [0xffff80ff7e4c2000]
   java.lang.Thread.State: TIMED_WAITING (on object monitor)
    at java.lang.Object.wait(Native Method)
    - waiting on <0x00000007d3e817b8> (a org.apache.commons.pool.impl.GenericObjectPool$Latch)
    at org.apache.commons.pool.impl.GenericObjectPool.borrowObject(GenericObjectPool.java:1123)
    - locked <0x00000007d3e817b8> (a org.apache.commons.pool.impl.GenericObjectPool$Latch)
    at org.apache.commons.dbcp.PoolingDataSource.getConnection(PoolingDataSource.java:96)
    at org.apache.commons.dbcp.BasicDataSource.getConnection(BasicDataSource.java:880)
    at org.sakaiproject.db.impl.BasicSqlService.borrowConnection(BasicSqlService.java:260)
    at org.sakaiproject.db.impl.BasicSqlService.dbRead(BasicSqlService.java:540)
    at org.sakaiproject.db.impl.BasicSqlService.dbRead(BasicSqlService.java:471)
    at org.sakaiproject.authz.impl.DbAuthzGroupService$DbStorage.isAllowed(DbAuthzGroupService.java:1604)
    at org.sakaiproject.authz.impl.BaseAuthzGroupService.isAllowed(BaseAuthzGroupService.java:871)
    at org.sakaiproject.authz.impl.SakaiSecurity.isSuperUser(SakaiSecurity.java:201)
    at org.sakaiproject.authz.impl.SakaiSecurity.unlock(SakaiSecurity.java:261)
    at org.sakaiproject.authz.impl.SakaiSecurity.unlock(SakaiSecurity.java:245)
    at org.sakaiproject.authz.impl.SakaiSecurity.unlock(SakaiSecurity.java:237)
    at org.sakaiproject.authz.impl.SakaiSecurity.unlock(SakaiSecurity.java:223)
    at org.sakaiproject.site.impl.BaseSiteService.unlockCheck(BaseSiteService.java:190)
    at org.sakaiproject.site.impl.BaseSiteService.unlock(BaseSiteService.java:210)
    at org.sakaiproject.site.impl.BaseSiteService.allowAccessSite(BaseSiteService.java:560)
    at org.sakaiproject.site.impl.BaseSiteService.getSiteVisit(BaseSiteService.java:790)
    at org.sakaiproject.site.cover.SiteService.getSiteVisit(SiteService.java:140)
    at org.sakaiproject.portal.charon.site.PortalSiteHelperImpl.getSiteVisit(PortalSiteHelperImpl.java:963)
    at org.sakaiproject.portal.charon.handlers.SiteHandler.doSite(SiteHandler.java:213)
    at org.sakaiproject.portal.charon.handlers.SiteHandler.doGet(SiteHandler.java:135)
    at org.sakaiproject.portal.charon.SkinnableCharonPortal.doGet(SkinnableCharonPortal.java:894)
    at javax.servlet.http.HttpServlet.service(HttpServlet.java:621)
    at javax.servlet.http.HttpServlet.service(HttpServlet.java:728)
    at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:305)
    at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:210)
    at org.sakaiproject.util.RequestFilter.doFilter(RequestFilter.java:695)
    at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:243)
    at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:210)
    at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:222)
    at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:123)
    at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:472)
    at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:171)
    at org.apache.catalina.ha.session.JvmRouteBinderValve.invoke(JvmRouteBinderValve.java:219)
    at org.apache.catalina.ha.tcp.ReplicationValve.invoke(ReplicationValve.java:335)
    at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:99)
    at org.apache.catalina.valves.AccessLogValve.invoke(AccessLogValve.java:953)
    at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:118)
    at org.apache.catalina.ha.tcp.ReplicationValve.invoke(ReplicationValve.java:335)
    at org.apache.catalina.ha.session.JvmRouteBinderValve.invoke(JvmRouteBinderValve.java:219)
    at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:408)
    at org.apache.coyote.http11.AbstractHttp11Processor.process(AbstractHttp11Processor.java:1023)
    at org.apache.coyote.AbstractProtocol$AbstractConnectionHandler.process(AbstractProtocol.java:589)
    at org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.run(NioEndpoint.java:1686)
    - locked <0x00000007e02f4da8> (a org.apache.tomcat.util.net.NioChannel)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
    at java.lang.Thread.run(Thread.java:722)

<table width="100%" border="0" cellspacing="0" cellpadding="0" style="width:100%;"> 
<tr>
<td align="left" style="text-align:justify;"><font face="arial,sans-serif" size="1" color="#999999"><span style="font-size:11px;">This communication is intended for the addressee only. It is confidential. If you have received this communication in error, please notify us immediately and destroy the original message. You may not copy or disseminate this communication without the permission of the University. Only authorised signatories are competent to enter into agreements on behalf of the University and recipients are thus advised that the content of this message may not be legally binding on the University and may contain the personal views and opinions of the author, which are not necessarily the views and opinions of The University of the Witwatersrand, Johannesburg. All agreements between the University and outsiders are subject to South African Law unless the University agrees in writing to the contrary. </span></font></td>
</tr>
</table>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://collab.sakaiproject.org/pipermail/sakai-dev/attachments/20140325/4bd0ad29/attachment.html 