[Deploying Sakai] dbcp.SQLNestedException: Cannot get a connection, pool error Timeout waiting for idle object

[Martin B. Smith]

Howdy Production list --

I'm wondering if someone can point me in the right direction with a
problem we've seen on one of our application nodes during & after load
testing.

We're seeing what appears to be DBCP exhausting connections in the pool,
and then *never recovering after the load is done*:

Caused by: java.util.NoSuchElementException: Timeout waiting for idle
object, at:
org.apache.commons.pool.impl.GenericObjectPool.borrowObject(GenericObjectPool.java:825)
org.apache.commons.dbcp.PoolingDataSource.getConnection(PoolingDataSource.java:96)

Now, this is from the 2.7 branch and using Oracle, and the only things
happening are a site stats 'tool activity' report and an assignment
submission (500 users concurrently, about half doing each activity).
This problem persists even after I can see that Oracle has plenty of
available room to establish new connections.

Even after all of this load subsides, it still can't get connections out
of the pool. We're seeing this when we only have one application node
for all 500 users. When I add more, I don't see a single node get into
this stuck state.

I've found the following Sakai bugs:
 http://jira.sakaiproject.org/browse/SAK-11852
 http://jira.sakaiproject.org/browse/SAK-8434
 http://jira.sakaiproject.org/browse/SAK-11877

And I've found mailing list discussions about this exception:
 http://bit.ly/8ZOMwC
 http://bit.ly/bAKeyz
 http://bit.ly/c4FJDL

Am I running into SAK-11852, a.k.a. "DBCP not recovering from db being
temporarily unavailable"? Are there some cache settings I really should
be setting higher, such as
cacheMinutes at org.sakaiproject.site.api.SiteService?

I do see mentioned in a Sakai developer blog (http://bit.ly/byYW06)
that, 'Mich just tuned off "evictions" and "didn't let the pool cycle"'.
Michigan, do you have any particular recommendations in that area?

Can someone hit me with a clue, if one exists? :)

Thanks in advance,
-- 
Martin B. Smith
smithmb at ufl.edu - (352) 273-1374
CNS/Open Systems Group
University of Florida

-------------- next part --------------
A non-text attachment was scrubbed...
Name: smime.p7s
Type: application/pkcs7-signature
Size: 5497 bytes
Desc: S/MIME Cryptographic Signature
Url : http://collab.sakaiproject.org/pipermail/production/attachments/20100426/da8a6fe2/attachment-0001.bin