[Building Sakai] tomcat 7 threads maxing out , freezing Sakai instance

[Matthew Buckett]

On 25 March 2014 07:12, David Wafula Wanyonyi <
DavidWafula.Wanyonyi at wits.ac.za> wrote:

>  Dear all,
> Lately our 2.9.2 Sakai  instances periodically grind to a halt atleast
> once per day. We have found that when this happens, tomcat will have maxed
> out the thread count. The DB writes fail from tomcat logs, with the
> following entry in logs:
>
> ...
> Caused by: org.apache.commons.dbcp.SQLNestedException: Cannot get a
> connection, pool error Timeout waiting for idle object
>         at
> org.apache.commons.dbcp.PoolingDataSource.getConnection(PoolingDataSource.java:104)
> ...
>
> Only a restart gets things going.
>

Are all your tomcat workers grinding to a halt at the same time? If yes
then as you suggest in your next message the connection to the database
looks like the culprit.

We tried to increase maxThread counts in tomcat conf, but this did not
> really help, as eventually they were all used up. We switched to nio
> connector in tomcat's serve.xml as follows:
>
>  <Connector port="8080"
> protocol="org.apache.coyote.http11.Http11NioProtocol"
>                connectionTimeout="20000"
>                redirectPort="8443" maxThreads="250"/>
>
> Other info:
> -----------------
> Server version: Apache Tomcat/7.0.41
> Server number:  7.0.41.0
> OS Name:        SunOS
> OS Version:     5.11
> Architecture:   x86 (we actually run on 64bit..not sure why it give this)
> JVM Version:    1.7.0_21-b11
>
> Any help on this will be appreciated.
>
> This seems to hold on longer, but eventually it still runs out.   I have
> attached a full thread dump, but here is an excerpt of where things seem to
> go wrong:
>

I can't see the full thread dump.

-- 
  Matthew Buckett, VLE Developer, IT Services, University of Oxford
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://collab.sakaiproject.org/pipermail/sakai-dev/attachments/20140325/7dfd0985/attachment.html