[Building Sakai] tomcat 7 threads maxing out , freezing Sakai instance

[David Wafula Wanyonyi]

Thanks everyone for the input. We finally tracked down the code we deployed recently that was not closing connections. This was becoming such a headache. So relieved !!!!!  Things seem to have settled down now.
We have been using http://java-monitor.com/ to monitor the nodes.
regards.

________________________________________
From: John Bush [jbush at anisakai.com]
Sent: Tuesday, March 25, 2014 6:47 PM
To: Sam Ottenhoff
Cc: David Wafula Wanyonyi; dev sakai
Subject: Re: [Building Sakai] tomcat 7 threads maxing out , freezing Sakai instance

Yeah, just to underscore what Sam said b/c I've had to chase down a
lot of these types of problems.  When the system is under strain, you
almost always will start to see contention around db connections,
because that is the resource that is used more than anything else.
Its usually (not always) just a symptom, not the cause.

Also consider deploying the free version of
http://www.appdynamics.com/.  It only keeps 2 hrs of history, and
won't show you everything the enterprise version does, but in a lot of
cases its enough to pin point things.   I've found this to be a really
great root cause detection tool you can use in production with very
little overhead.

On Tue, Mar 25, 2014 at 6:02 AM, Sam Ottenhoff <ottenhoff at longsight.com> wrote:
>>
>> Lately our 2.9.2 Sakai  instances periodically grind to a halt atleast
>> once per day. We have found that when this happens, tomcat will have maxed
>> out the thread count.
>
>
>
> Are you connecting to the Tomcat instance's JMX port via VisualVM and
> tracking threads/memory usage/GCs?  What do the charts look like as your
> instance is maxed out?
>
>
>>
>> The DB writes fail from tomcat logs, with the following entry in logs:
>>
>> ...
>> Caused by: org.apache.commons.dbcp.SQLNestedException: Cannot get a
>> connection, pool error Timeout waiting for idle object
>>         at
>> org.apache.commons.dbcp.PoolingDataSource.getConnection(PoolingDataSource.java:104)
>
>
>
> So this is either the root of the problem or a side effect of the actual
> problem.  Do you have any custom code running on this instance that uses
> database pool connections?  Do you have any lightly tested community code
> running on your instance that accesses a database pool?  If yes, I would
> begin a code review and make sure that the pool connections are being
> returned properly in a { finally } block.
>
> But this error could also be a side effect of your Tomcat instance being
> OutOfMemory or out of threads.  Do you see any warnings about OutOfMemory in
> the logs?
>
>
>
>>
>> We tried to increase maxThread counts in tomcat conf, but this did not
>> really help, as eventually they were all used up. We switched to nio
>> connector in tomcat's serve.xml as follows:
>>
>>  <Connector port="8080"
>> protocol="org.apache.coyote.http11.Http11NioProtocol"
>>                connectionTimeout="20000"
>>                redirectPort="8443" maxThreads="250"/>
>
>
>
> Yeah, I would not modify the default connectors.  They should be fine.  You
> need to figure out either a) why the database pool is being exhausted or b)
> why the JVM's allocated resources are being exhausted.
>
>
>
> _______________________________________________
> sakai-dev mailing list
> sakai-dev at collab.sakaiproject.org
> http://collab.sakaiproject.org/mailman/listinfo/sakai-dev
>
> TO UNSUBSCRIBE: send email to sakai-dev-unsubscribe at collab.sakaiproject.org
> with a subject of "unsubscribe"



--
John Bush
602-490-0470

** This message is neither private nor confidential in fact the US
government is storing it in a warehouse located in Utah for future
data mining use cases should they arise. **
=
<table width="100%" border="0" cellspacing="0" cellpadding="0" style="width:100%;"> 
<tr>
<td align="left" style="text-align:justify;"><font face="arial,sans-serif" size="1" color="#999999"><span style="font-size:11px;">This communication is intended for the addressee only. It is confidential. If you have received this communication in error, please notify us immediately and destroy the original message. You may not copy or disseminate this communication without the permission of the University. Only authorised signatories are competent to enter into agreements on behalf of the University and recipients are thus advised that the content of this message may not be legally binding on the University and may contain the personal views and opinions of the author, which are not necessarily the views and opinions of The University of the Witwatersrand, Johannesburg. All agreements between the University and outsiders are subject to South African Law unless the University agrees in writing to the contrary. </span></font></td>
</tr>
</table>