[Building Sakai] tomcat 7 threads maxing out , freezing Sakai instance

[Sam Ottenhoff]

>
>
> Lately our 2.9.2 Sakai  instances periodically grind to a halt atleast
> once per day. We have found that when this happens, tomcat will have maxed
> out the thread count.
>


Are you connecting to the Tomcat instance's JMX port via VisualVM and
tracking threads/memory usage/GCs?  What do the charts look like as your
instance is maxed out?



> The DB writes fail from tomcat logs, with the following entry in logs:
>
> ...
> Caused by: org.apache.commons.dbcp.SQLNestedException: Cannot get a
> connection, pool error Timeout waiting for idle object
>         at
> org.apache.commons.dbcp.PoolingDataSource.getConnection(PoolingDataSource.java:104)
>


So this is either the root of the problem or a side effect of the actual
problem.  Do you have any custom code running on this instance that uses
database pool connections?  Do you have any lightly tested community code
running on your instance that accesses a database pool?  If yes, I would
begin a code review and make sure that the pool connections are being
returned properly in a { finally } block.

But this error could also be a side effect of your Tomcat instance being
OutOfMemory or out of threads.  Do you see any warnings about OutOfMemory
in the logs?




> We tried to increase maxThread counts in tomcat conf, but this did not
> really help, as eventually they were all used up. We switched to nio
> connector in tomcat's serve.xml as follows:
>
>  <Connector port="8080"
> protocol="org.apache.coyote.http11.Http11NioProtocol"
>                connectionTimeout="20000"
>                redirectPort="8443" maxThreads="250"/>
>


Yeah, I would not modify the default connectors.  They should be fine.  You
need to figure out either a) why the database pool is being exhausted or b)
why the JVM's allocated resources are being exhausted.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://collab.sakaiproject.org/pipermail/sakai-dev/attachments/20140325/1b77abf1/attachment.html