[Deploying Sakai] possible dbcp issue

[Charles Hedrick]

what versions of dbcp and pool are you using? It looks like dbcp 1.2.2  
is the current released version, but most of the deadlocks are fixed  
in 1.3. pool 1.5.2 seems to be current, and earlier versions are all  
highly buggy. 1.5.2 is claimed to be compatible with 1.3, so we should  
be able to switch to it.

The problem with validation on return to pool is that there's a  
deadlock that it will trigger. We had to remove it to get stable  
operation. That was the cause of the odd GC pauses we saw for a long  
time. That may be fixed in pool 1.4. (The problem is that the  
validation query is done during the process of allocating/deallocating  
a connection. Before pool 1.4 it's done while the major lock is still  
held. This can produce a deadlock, and even when it doesn't it will  
slow things down a lot because it effectively serializes database I/O.)

I'm probably going to go to 1.3/1.5.2 on our test system and then move  
it into production one front end at a time. If that doesn't fix the  
problem, I'll enable removeAbandoned. In the meantime I have a JSP  
that can create new connections for the pool, which I'll use if the  
number of idle connections gets dangerously low. This problem seems to  
be new with Sakai 2.6, which I'm guessing means that it's new with  
dbcp 1.2.2/pool 1.3.

I looked at removeabandoned. What bothers me about it is that it  
doesn't trigger until you're very near to running out of connections.  
That means that we won't see whether it's working until we get into a  
situation that I really don't want to get into.


On Sep 10, 2009, at 9:44 AM, Jason Shao (CampusEAI Consortium) wrote:

> I saw similar issues in dbcp in a tomcat DataSource (not Sakai)  
> running in tomcat 5.5.27 and JDK 1.6.0_13 and MySQL 5.0.67. In the  
> short term we've added both a validationQuery (on return to the  
> pool) and activated "removeAbandoned" to try and see if it's a leak/ 
> retained connection (though it doesn't show up on either Netstat or  
> in MySQL using "show full processlist";
>
> In the short term the removeAbandoned seems to have eliminated the  
> ghost connections, and we've turned on JMX to give better longer- 
> term stats while we watch. The ghost connections don't seem to be  
> gradual - we run for quite a while, then all of a sudden see a block  
> of 20 or so appear -- could be traffic/load/application behavior  
> related though.
>
> We've reset our settings to (sanitized):
>
> 54        <Resource
> 55             name="XXXXXXXX"
> 56             auth="Container"
> 57             type="javax.sql.DataSource"
> 58             driverClassName="com.mysql.jdbc.Driver"
> 59              
> factory="org.apache.tomcat.dbcp.dbcp.BasicDataSourceFactory"
> 60             url="jdbc:mysql://localhost/XXXXXXX? 
> useUnicode=true&amp;characterEncoding=UTF-8"
> 61             username="XXXXXXX"
> 62             password="XXXXXXX"
> 63             maxActive="100"
> 64             validationQuery="select '1'"
> 65             testOnBorrow="false"
> 66             testOnReturn="true"
> 67             testWhileIdle="true"
> 68             timeBetweenEvictionRunsMillis="2400000"
> 69             maxWait="30000"
> 70             removeAbandonedTimeout="30"
> 71             removeAbandoned="true"
> 72             logAbandoned="true"
> 73           />
>
> --
> Jason Shao
> Director of Product Development
> CampusEAI Consortium
> 1940 East 6th Street, 11th Floor
> Cleveland, OH 44114
> Tel: 216.589.9626x249
> Fax: 216.589.9639
>
> -----Original Message-----
> From: production-bounces at collab.sakaiproject.org [mailto:production- 
> bounces at collab.sakaiproject.org] On Behalf Of Seth Theriault
> Sent: Thursday, September 10, 2009 8:23 AM
> To: Charles Hedrick
> Cc: production at collab.sakaiproject.org
> Subject: Re: [Deploying Sakai] possible dbcp issue
>
> Charles Hedrick wrote:
>
>> Is it safe to change versions of dbcp and pool? I'm
>> concerned because there do seem to be at least some
>> incompatibilities.
>
> We are already running the latest dbcp (1.2.2) from 2007.
>
> I suppose that there could be issues with commons-pool since
> it has been updated more recently. I guess some testing is in
> order if your issues are addressed with a newer commons-pool.
>
> Seth
>
> _______________________________________________
> production mailing list
> production at collab.sakaiproject.org
> http://collab.sakaiproject.org/mailman/listinfo/production
>
> TO UNSUBSCRIBE: send email to production-unsubscribe at collab.sakaiproject.org 
>  with a subject of "unsubscribe"
>
> Your input is important to improve upon our continuous efforts to  
> service you better. Please e-mail my manager at anjli_jain at campuseai.org 
>  with any feedback.
>
> CONFIDENTIALITY NOTICE:
> This e-mail together with any attachments is proprietary and  
> confidential; intended for only the recipient(s) named above and may  
> contain information that is privileged. You should not retain, copy  
> or use this e-mail or any attachments for any purpose, or disclose  
> all or any part of the contents to any person. Any views or opinions  
> expressed in this e-mail are those of the author and do not  
> represent those of CampusEAI Consortium or the Open Student  
> Television Network. If you have received this e-mail in error, or  
> are not the named recipient(s), you are hereby notified that any  
> review, dissemination, distribution or copying of this communication  
> is prohibited by the sender and to do so might constitute a  
> violation of the Electronic Communications Privacy Act, 18 U.S.C.  
> section 2510-2521. Please immediately notify the sender and delete  
> this e-mail and any attachments from your computer. Warning:  
> Although precautions have been taken to make sure no viruses are  
> present in this e-mail, the companies cannot accept responsibility  
> for any loss or damage that arise from the use of this e-mail or  
> attachments.