[Deploying Sakai] possible dbcp issue

Jason Shao (CampusEAI Consortium) jason_shao at campuseai.org
Thu Sep 10 07:23:33 PDT 2009


We're using 1.2.2 (as distributed by Liferay - I believe there's no changes).

Yes, we view remove abandoned more as last ditch diagnostics - though it does have the advantage of generating stack traces when connections are opened or new statements are triggered, which if you are running in a debugger or certain profilers you can view at runtime to see which connections were acquired by what piece of code.

I have *very rarely* used conditional exceptions in Eclipse on running systems (though the couple of times I did it was a lifesaver) - though heap walking of snapshots can glean useful data as well. Admittedly a 13GB heap may require an obscene amount of time to take a snapshot - though jmap's hprof format is fairly fast (seems to be mostly limited on our systems by disk IO) - can still take upwards of 5-10 min for a 3GB (64-bit) heap. Yourkit's heap format has slightly more data (and they have the free open-source licenses, not sure if Sakai has registered as a project for one - Jasig has some floating around).

Jason

--
Jason Shao
Director of Product Development
CampusEAI Consortium
1940 East 6th Street, 11th Floor
Cleveland, OH 44114
Tel: 216.589.9626x249
Fax: 216.589.9639


-----Original Message-----
From: Charles Hedrick [mailto:hedrick at rutgers.edu]
Sent: Thursday, September 10, 2009 10:10 AM
To: Jason Shao (CampusEAI Consortium)
Cc: 'Seth Theriault'; production at collab.sakaiproject.org
Subject: Re: [Deploying Sakai] possible dbcp issue

what versions of dbcp and pool are you using? It looks like dbcp 1.2.2
is the current released version, but most of the deadlocks are fixed
in 1.3. pool 1.5.2 seems to be current, and earlier versions are all
highly buggy. 1.5.2 is claimed to be compatible with 1.3, so we should
be able to switch to it.

The problem with validation on return to pool is that there's a
deadlock that it will trigger. We had to remove it to get stable
operation. That was the cause of the odd GC pauses we saw for a long
time. That may be fixed in pool 1.4. (The problem is that the
validation query is done during the process of allocating/deallocating
a connection. Before pool 1.4 it's done while the major lock is still
held. This can produce a deadlock, and even when it doesn't it will
slow things down a lot because it effectively serializes database I/O.)

I'm probably going to go to 1.3/1.5.2 on our test system and then move
it into production one front end at a time. If that doesn't fix the
problem, I'll enable removeAbandoned. In the meantime I have a JSP
that can create new connections for the pool, which I'll use if the
number of idle connections gets dangerously low. This problem seems to
be new with Sakai 2.6, which I'm guessing means that it's new with
dbcp 1.2.2/pool 1.3.

I looked at removeabandoned. What bothers me about it is that it
doesn't trigger until you're very near to running out of connections.
That means that we won't see whether it's working until we get into a
situation that I really don't want to get into.


On Sep 10, 2009, at 9:44 AM, Jason Shao (CampusEAI Consortium) wrote:

> I saw similar issues in dbcp in a tomcat DataSource (not Sakai)
> running in tomcat 5.5.27 and JDK 1.6.0_13 and MySQL 5.0.67. In the
> short term we've added both a validationQuery (on return to the
> pool) and activated "removeAbandoned" to try and see if it's a leak/
> retained connection (though it doesn't show up on either Netstat or
> in MySQL using "show full processlist";
>
> In the short term the removeAbandoned seems to have eliminated the
> ghost connections, and we've turned on JMX to give better longer-
> term stats while we watch. The ghost connections don't seem to be
> gradual - we run for quite a while, then all of a sudden see a block
> of 20 or so appear -- could be traffic/load/application behavior
> related though.
>
> We've reset our settings to (sanitized):
>
> 54        <Resource
> 55             name="XXXXXXXX"
> 56             auth="Container"
> 57             type="javax.sql.DataSource"
> 58             driverClassName="com.mysql.jdbc.Driver"
> 59
> factory="org.apache.tomcat.dbcp.dbcp.BasicDataSourceFactory"
> 60             url="jdbc:mysql://localhost/XXXXXXX?
> useUnicode=true&amp;characterEncoding=UTF-8"
> 61             username="XXXXXXX"
> 62             password="XXXXXXX"
> 63             maxActive="100"
> 64             validationQuery="select '1'"
> 65             testOnBorrow="false"
> 66             testOnReturn="true"
> 67             testWhileIdle="true"
> 68             timeBetweenEvictionRunsMillis="2400000"
> 69             maxWait="30000"
> 70             removeAbandonedTimeout="30"
> 71             removeAbandoned="true"
> 72             logAbandoned="true"
> 73           />
>
> --
> Jason Shao
> Director of Product Development
> CampusEAI Consortium
> 1940 East 6th Street, 11th Floor
> Cleveland, OH 44114
> Tel: 216.589.9626x249
> Fax: 216.589.9639
>
> -----Original Message-----
> From: production-bounces at collab.sakaiproject.org [mailto:production-
> bounces at collab.sakaiproject.org] On Behalf Of Seth Theriault
> Sent: Thursday, September 10, 2009 8:23 AM
> To: Charles Hedrick
> Cc: production at collab.sakaiproject.org
> Subject: Re: [Deploying Sakai] possible dbcp issue
>
> Charles Hedrick wrote:
>
>> Is it safe to change versions of dbcp and pool? I'm
>> concerned because there do seem to be at least some
>> incompatibilities.
>
> We are already running the latest dbcp (1.2.2) from 2007.
>
> I suppose that there could be issues with commons-pool since
> it has been updated more recently. I guess some testing is in
> order if your issues are addressed with a newer commons-pool.
>
> Seth
>
> _______________________________________________
> production mailing list
> production at collab.sakaiproject.org
> http://collab.sakaiproject.org/mailman/listinfo/production
>
> TO UNSUBSCRIBE: send email to production-unsubscribe at collab.sakaiproject.org
>  with a subject of "unsubscribe"
>
> Your input is important to improve upon our continuous efforts to
> service you better. Please e-mail my manager at anjli_jain at campuseai.org
>  with any feedback.
>
> CONFIDENTIALITY NOTICE:
> This e-mail together with any attachments is proprietary and
> confidential; intended for only the recipient(s) named above and may
> contain information that is privileged. You should not retain, copy
> or use this e-mail or any attachments for any purpose, or disclose
> all or any part of the contents to any person. Any views or opinions
> expressed in this e-mail are those of the author and do not
> represent those of CampusEAI Consortium or the Open Student
> Television Network. If you have received this e-mail in error, or
> are not the named recipient(s), you are hereby notified that any
> review, dissemination, distribution or copying of this communication
> is prohibited by the sender and to do so might constitute a
> violation of the Electronic Communications Privacy Act, 18 U.S.C.
> section 2510-2521. Please immediately notify the sender and delete
> this e-mail and any attachments from your computer. Warning:
> Although precautions have been taken to make sure no viruses are
> present in this e-mail, the companies cannot accept responsibility
> for any loss or damage that arise from the use of this e-mail or
> attachments.


Your input is important to improve upon our continuous efforts to service you better. Please e-mail my manager at anjli_jain at campuseai.org with any feedback.

CONFIDENTIALITY NOTICE:
This e-mail together with any attachments is proprietary and confidential; intended for only the recipient(s) named above and may contain information that is privileged. You should not retain, copy or use this e-mail or any attachments for any purpose, or disclose all or any part of the contents to any person. Any views or opinions expressed in this e-mail are those of the author and do not represent those of CampusEAI Consortium or the Open Student Television Network. If you have received this e-mail in error, or are not the named recipient(s), you are hereby notified that any review, dissemination, distribution or copying of this communication is prohibited by the sender and to do so might constitute a violation of the Electronic Communications Privacy Act, 18 U.S.C. section 2510-2521. Please immediately notify the sender and delete this e-mail and any attachments from your computer. Warning: Although precautions have been taken to make sure no viruses are present in this e-mail, the companies cannot accept responsibility for any loss or damage that arise from the use of this e-mail or attachments.


More information about the production mailing list