[Building Sakai] tomcat 7 threads maxing out , freezing Sakai instance

[John Bush]

Yeah, just to underscore what Sam said b/c I've had to chase down a
lot of these types of problems.  When the system is under strain, you
almost always will start to see contention around db connections,
because that is the resource that is used more than anything else.
Its usually (not always) just a symptom, not the cause.

Also consider deploying the free version of
http://www.appdynamics.com/.  It only keeps 2 hrs of history, and
won't show you everything the enterprise version does, but in a lot of
cases its enough to pin point things.   I've found this to be a really
great root cause detection tool you can use in production with very
little overhead.

On Tue, Mar 25, 2014 at 6:02 AM, Sam Ottenhoff <ottenhoff at longsight.com> wrote:
>>
>> Lately our 2.9.2 Sakai  instances periodically grind to a halt atleast
>> once per day. We have found that when this happens, tomcat will have maxed
>> out the thread count.
>
>
>
> Are you connecting to the Tomcat instance's JMX port via VisualVM and
> tracking threads/memory usage/GCs?  What do the charts look like as your
> instance is maxed out?
>
>
>>
>> The DB writes fail from tomcat logs, with the following entry in logs:
>>
>> ...
>> Caused by: org.apache.commons.dbcp.SQLNestedException: Cannot get a
>> connection, pool error Timeout waiting for idle object
>>         at
>> org.apache.commons.dbcp.PoolingDataSource.getConnection(PoolingDataSource.java:104)
>
>
>
> So this is either the root of the problem or a side effect of the actual
> problem.  Do you have any custom code running on this instance that uses
> database pool connections?  Do you have any lightly tested community code
> running on your instance that accesses a database pool?  If yes, I would
> begin a code review and make sure that the pool connections are being
> returned properly in a { finally } block.
>
> But this error could also be a side effect of your Tomcat instance being
> OutOfMemory or out of threads.  Do you see any warnings about OutOfMemory in
> the logs?
>
>
>
>>
>> We tried to increase maxThread counts in tomcat conf, but this did not
>> really help, as eventually they were all used up. We switched to nio
>> connector in tomcat's serve.xml as follows:
>>
>>  <Connector port="8080"
>> protocol="org.apache.coyote.http11.Http11NioProtocol"
>>                connectionTimeout="20000"
>>                redirectPort="8443" maxThreads="250"/>
>
>
>
> Yeah, I would not modify the default connectors.  They should be fine.  You
> need to figure out either a) why the database pool is being exhausted or b)
> why the JVM's allocated resources are being exhausted.
>
>
>
> _______________________________________________
> sakai-dev mailing list
> sakai-dev at collab.sakaiproject.org
> http://collab.sakaiproject.org/mailman/listinfo/sakai-dev
>
> TO UNSUBSCRIBE: send email to sakai-dev-unsubscribe at collab.sakaiproject.org
> with a subject of "unsubscribe"



-- 
John Bush
602-490-0470

** This message is neither private nor confidential in fact the US
government is storing it in a warehouse located in Utah for future
data mining use cases should they arise. **