[Building Sakai] tomcat 7 threads maxing out , freezing Sakai instance
David Wafula Wanyonyi
DavidWafula.Wanyonyi at wits.ac.za
Tue Mar 25 01:04:35 PDT 2014
It seems to me the issue stems from these hardcoded values in /sakai-2.9.2-all/kernel/kernel-component/src/main/webapp/WEB-INF/db-components.xml
I can't find where they are set dynamically.
......
<!-- The initial number of connections that are created when the pool is started. -->
<property name="initialSize">
<value>10</value>
</property>
<!-- The maximum number of active connections that can be allocated from this pool at the same time, or negative for no limit. -->
<property name="maxActive">
<value>50</value>
</property>
<!-- The maximum number of active connections that can remain idle in the pool, without extra ones being released, or negative for no limit. -->
<property name="maxIdle">
<value>10</value>
</property>
<!-- The minimum number of active connections that can remain idle in the pool, without extra ones being created, or 0 to create none. -->
<property name="minIdle">
<value>0</value>
</property>
<!-- The maximum number of milliseconds that the pool will wait (when there
are no available connections) for a connection to be returned before
throwing an exception, or -1 to wait indefinitely. -->
<property name="maxWait">
<value>300000</value>
</property>
<!-- The number of objects to examine during each run of the idle object
evictor thread (if any). -->
<property name="numTestsPerEvictionRun">
<value>3</value>
</property>
.....
________________________________
From: David Wafula Wanyonyi [DavidWafula.Wanyonyi at wits.ac.za]
Sent: Tuesday, March 25, 2014 9:12 AM
To: dev sakai
Subject: [Building Sakai] tomcat 7 threads maxing out , freezing Sakai instance
Dear all,
Lately our 2.9.2 Sakai instances periodically grind to a halt atleast once per day. We have found that when this happens, tomcat will have maxed out the thread count. The DB writes fail from tomcat logs, with the following entry in logs:
...
Caused by: org.apache.commons.dbcp.SQLNestedException: Cannot get a connection, pool error Timeout waiting for idle object
at org.apache.commons.dbcp.PoolingDataSource.getConnection(PoolingDataSource.java:104)
...
Only a restart gets things going.
We tried to increase maxThread counts in tomcat conf, but this did not really help, as eventually they were all used up. We switched to nio connector in tomcat's serve.xml as follows:
<Connector port="8080" protocol="org.apache.coyote.http11.Http11NioProtocol"
connectionTimeout="20000"
redirectPort="8443" maxThreads="250"/>
Other info:
-----------------
Server version: Apache Tomcat/7.0.41
Server number: 7.0.41.0
OS Name: SunOS
OS Version: 5.11
Architecture: x86 (we actually run on 64bit..not sure why it give this)
JVM Version: 1.7.0_21-b11
Any help on this will be appreciated.
This seems to hold on longer, but eventually it still runs out. I have attached a full thread dump, but here is an excerpt of where things seem to go wrong:
"http-nio-8080-exec-487" daemon prio=3 tid=0x0000000007dc0800 nid=0x25b4 in Object.wait() [0xffff80ff7e4c2000]
java.lang.Thread.State: TIMED_WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
- waiting on <0x00000007d3e817b8> (a org.apache.commons.pool.impl.GenericObjectPool$Latch)
at org.apache.commons.pool.impl.GenericObjectPool.borrowObject(GenericObjectPool.java:1123)
- locked <0x00000007d3e817b8> (a org.apache.commons.pool.impl.GenericObjectPool$Latch)
at org.apache.commons.dbcp.PoolingDataSource.getConnection(PoolingDataSource.java:96)
at org.apache.commons.dbcp.BasicDataSource.getConnection(BasicDataSource.java:880)
at org.sakaiproject.db.impl.BasicSqlService.borrowConnection(BasicSqlService.java:260)
at org.sakaiproject.db.impl.BasicSqlService.dbRead(BasicSqlService.java:540)
at org.sakaiproject.db.impl.BasicSqlService.dbRead(BasicSqlService.java:471)
at org.sakaiproject.authz.impl.DbAuthzGroupService$DbStorage.isAllowed(DbAuthzGroupService.java:1604)
at org.sakaiproject.authz.impl.BaseAuthzGroupService.isAllowed(BaseAuthzGroupService.java:871)
at org.sakaiproject.authz.impl.SakaiSecurity.isSuperUser(SakaiSecurity.java:201)
at org.sakaiproject.authz.impl.SakaiSecurity.unlock(SakaiSecurity.java:261)
at org.sakaiproject.authz.impl.SakaiSecurity.unlock(SakaiSecurity.java:245)
at org.sakaiproject.authz.impl.SakaiSecurity.unlock(SakaiSecurity.java:237)
at org.sakaiproject.authz.impl.SakaiSecurity.unlock(SakaiSecurity.java:223)
at org.sakaiproject.site.impl.BaseSiteService.unlockCheck(BaseSiteService.java:190)
at org.sakaiproject.site.impl.BaseSiteService.unlock(BaseSiteService.java:210)
at org.sakaiproject.site.impl.BaseSiteService.allowAccessSite(BaseSiteService.java:560)
at org.sakaiproject.site.impl.BaseSiteService.getSiteVisit(BaseSiteService.java:790)
at org.sakaiproject.site.cover.SiteService.getSiteVisit(SiteService.java:140)
at org.sakaiproject.portal.charon.site.PortalSiteHelperImpl.getSiteVisit(PortalSiteHelperImpl.java:963)
at org.sakaiproject.portal.charon.handlers.SiteHandler.doSite(SiteHandler.java:213)
at org.sakaiproject.portal.charon.handlers.SiteHandler.doGet(SiteHandler.java:135)
at org.sakaiproject.portal.charon.SkinnableCharonPortal.doGet(SkinnableCharonPortal.java:894)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:621)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:728)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:305)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:210)
at org.sakaiproject.util.RequestFilter.doFilter(RequestFilter.java:695)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:243)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:210)
at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:222)
at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:123)
at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:472)
at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:171)
at org.apache.catalina.ha.session.JvmRouteBinderValve.invoke(JvmRouteBinderValve.java:219)
at org.apache.catalina.ha.tcp.ReplicationValve.invoke(ReplicationValve.java:335)
at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:99)
at org.apache.catalina.valves.AccessLogValve.invoke(AccessLogValve.java:953)
at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:118)
at org.apache.catalina.ha.tcp.ReplicationValve.invoke(ReplicationValve.java:335)
at org.apache.catalina.ha.session.JvmRouteBinderValve.invoke(JvmRouteBinderValve.java:219)
at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:408)
at org.apache.coyote.http11.AbstractHttp11Processor.process(AbstractHttp11Processor.java:1023)
at org.apache.coyote.AbstractProtocol$AbstractConnectionHandler.process(AbstractProtocol.java:589)
at org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.run(NioEndpoint.java:1686)
- locked <0x00000007e02f4da8> (a org.apache.tomcat.util.net.NioChannel)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:722)
This communication is intended
for the addressee only. It is confidential. If you have received this communication in error, please notify us immediately and destroy the original message. You may not copy or disseminate this communication without the permission of the University. Only authorised
signatories are competent to enter into agreements on behalf of the University and recipients are thus advised that the content of this message may not be legally binding on the University and may contain the personal views and opinions of the author, which
are not necessarily the views and opinions of The University of the Witwatersrand, Johannesburg. All agreements between the University and outsiders are subject to South African Law unless the University agrees in writing to the contrary.
<table width="100%" border="0" cellspacing="0" cellpadding="0" style="width:100%;">
<tr>
<td align="left" style="text-align:justify;"><font face="arial,sans-serif" size="1" color="#999999"><span style="font-size:11px;">This communication is intended for the addressee only. It is confidential. If you have received this communication in error, please notify us immediately and destroy the original message. You may not copy or disseminate this communication without the permission of the University. Only authorised signatories are competent to enter into agreements on behalf of the University and recipients are thus advised that the content of this message may not be legally binding on the University and may contain the personal views and opinions of the author, which are not necessarily the views and opinions of The University of the Witwatersrand, Johannesburg. All agreements between the University and outsiders are subject to South African Law unless the University agrees in writing to the contrary. </span></font></td>
</tr>
</table>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://collab.sakaiproject.org/pipermail/sakai-dev/attachments/20140325/293b32e0/attachment.html
More information about the sakai-dev
mailing list