[Deploying Sakai] [Building Sakai] Strange database connection memory issue at U-M

[Jason Shao (CampusEAI Consortium)]

c3p0 seems to have some interesting options:

softResetDefaultUser(), softReset( username, password ) and
softResetAllUsers()

For flushing/manipulating pools if you think something is leaking... And
looks like they're exposed as JMX Mbeans...

Jason

On 9/17/09 3:24 PM, "Noah Botimer" <botimer at umich.edu> wrote:

> I think we would do well to consider a simple work queue. We would
> push things that are expected to run long there. I'd say start with a
> generous estimate (10 sec at our scale, maybe?). Instrumenting the
> pool to know how long connections are held would help identify
> candidates to be taken out of immediate mode into queued mode. Some
> of these pieces would need asynchronous handling / notification at
> the UI.
>
> Once there, we could incrementally tune our attention threshold
> closer to the practical limits of instantaneous vs. queued work
> (maybe 2 seconds per thread?). As the set of things we'd break by
> imposing thread timeouts dwindled, we could consider being vicious to
> threads running unexpectedly long. With a reasonable, consistent
> infrastructure for deciding which mode something should use, the
> decision is much less daunting and the rules of "fast, queued, or
> killed" are tolerable.
>
> In this model, we would gain practical expertise in how to deal with
> failures more gracefully. I believe our current model is too
> optimistic, generally assuming complete success of every operation
> attempted, which adds brittleness to the app and data. It is worth
> studying each of the big cloud solutions here to see patterns and
> techniques.
>
> The important thing to note is that there is a very gentle ramp
> available here. A little tooling for deferred/notified work and
> tracking ill-behaved transactions wouldn't mandate immediate overhaul
> of all code. It would just give us a way to start moving toward a
> more resilient platform without the all-or-nothing paralysis that is
> easy to find in these topics.
>
> Please also note that this exercise need not be restricted to 2.x or
> 3.x. There are real scalability needs to consider on either platform
> and the basic techniques and wisdom (if not code) are transferrable.
>
> Thanks,
> -Noah
>
> On Sep 17, 2009, at 2:53 PM, Speelmon, Lance Day wrote:
>
>> Perhaps - probably the right place to start looking.  The problem is
>> that the tools to help troubleshoot these kind of leaks cannot be used
>> in production environments.  For example, if you tell the connection
>> pool to force return connections to the pool after a set amount of
>> time - it interferes with batch activities.  So, then the question
>> becomes: can these leaks be triggered in a test environment?  Or maybe
>> batch should be running out of a separate pool to allow this kind of
>> troubleshooting?  L
>>
>>
>> Lance Speelmon
>> Scholarly Technologist
>>
>> On Sep 17, 2009, at 2:31 PM, Seth Theriault wrote:
>>
>>> Sean DeMonner wrote:
>>>
>>>> We've wondered that too, but haven't found anything that is
>>>> running at those times. The jumps are also not consistently
>>>> spaced so we haven't seen a pattern.
>>>
>>> I am thinking that there is some new JDBC-related code in 2.6 and
>>> the database connections are not being closed properly (I guess
>>> in this case, it would mean releasing the conns back to the
>>> pool). I know that Michigan has the capacity to have a large sets
>>> of results returned from any query.
>>>
>>> Just a thought.
>>>
>>> Seth
>>>
>>> _______________________________________________
>>> production mailing list
>>> production at collab.sakaiproject.org
>>> http://collab.sakaiproject.org/mailman/listinfo/production
>>>
>>> TO UNSUBSCRIBE: send email to production-
>>> unsubscribe at collab.sakaiproject.org
>>>  with a subject of "unsubscribe"
>>
>> _______________________________________________
>> production mailing list
>> production at collab.sakaiproject.org
>> http://collab.sakaiproject.org/mailman/listinfo/production
>>
>> TO UNSUBSCRIBE: send email to production-
>> unsubscribe at collab.sakaiproject.org with a subject of "unsubscribe"
>>
>>
>
> _______________________________________________
> production mailing list
> production at collab.sakaiproject.org
> http://collab.sakaiproject.org/mailman/listinfo/production
>
> TO UNSUBSCRIBE: send email to production-unsubscribe at collab.sakaiproject.org
> with a subject of "unsubscribe"

--
Jason Shao
Director of Product Development
CampusEAI Consortium
1940 East 6th Street, 11th Floor
Cleveland, OH 44114
Tel: 216.589.9626x249
Fax: 216.589.9639


Your input is important to improve upon our continuous efforts to service you better. Please e-mail my manager at anjli_jain at campuseai.org with any feedback.

CONFIDENTIALITY NOTICE:
This e-mail together with any attachments is proprietary and confidential; intended for only the recipient(s) named above and may contain information that is privileged. You should not retain, copy or use this e-mail or any attachments for any purpose, or disclose all or any part of the contents to any person. Any views or opinions expressed in this e-mail are those of the author and do not represent those of CampusEAI Consortium or the Open Student Television Network. If you have received this e-mail in error, or are not the named recipient(s), you are hereby notified that any review, dissemination, distribution or copying of this communication is prohibited by the sender and to do so might constitute a violation of the Electronic Communications Privacy Act, 18 U.S.C. section 2510-2521. Please immediately notify the sender and delete this e-mail and any attachments from your computer. Warning: Although precautions have been taken to make sure no viruses are present in this e-mail, the companies cannot accept responsibility for any loss or damage that arise from the use of this e-mail or attachments.