[cle-release-team] [Building Sakai] Memory leak in Courier Service

[Maurer, Christopher Wayne]

I attached the "final" patch to the jira which is just slightly different than the last change in the code review.  Maybe this is something that could be quickly talked about on next week's release team call?  I won't be there but Greg should be able to talk about anything that was done.
The resolution is still set to "Won't fix" even though it was reopened.  Not sure that I have the ability to change that, but just an fyi for whenever it gets addressed!

Chris

From: <Maurer>, Christopher Maurer <chmaurer at iu.edu<mailto:chmaurer at iu.edu>>
Date: Wednesday, September 25, 2013 9:51 AM
To: dev sakai <sakai-dev at collab.sakaiproject.org<mailto:sakai-dev at collab.sakaiproject.org>>
Cc: "cle-release-team at collab.sakaiproject.org<mailto:cle-release-team at collab.sakaiproject.org>" <cle-release-team at collab.sakaiproject.org<mailto:cle-release-team at collab.sakaiproject.org>>
Subject: Re: [cle-release-team] [Building Sakai] Memory leak in Courier Service

https://crucible.sakaiproject.org/cru/SAKTRUNK-17

I've put together this code review on our proposed changes.  We're still doing testing, so there is the possibility of additional changes (just uploaded a new patch to it this morning, actually).  But please comment about anything that you see that might be making things worse rather than better!  I see that some people have already been in it looking around, so that's a good start.

Chris

From: Earle Nietzel <enietzel at anisakai.com<mailto:enietzel at anisakai.com>>
Date: Monday, September 23, 2013 1:48 PM
To: Christopher Maurer <chmaurer at iu.edu<mailto:chmaurer at iu.edu>>
Cc: Matthew Jones <matthew at longsight.com<mailto:matthew at longsight.com>>, Beth Kirschner <bkirschn at umich.edu<mailto:bkirschn at umich.edu>>, dev sakai <sakai-dev at collab.sakaiproject.org<mailto:sakai-dev at collab.sakaiproject.org>>
Subject: Re: [Building Sakai] Memory leak in Courier Service

Hi Chris

I would reopen it and attach the patch.


On Mon, Sep 23, 2013 at 10:29 AM, Maurer, Christopher Wayne <chmaurer at iu.edu<mailto:chmaurer at iu.edu>> wrote:
I'm working on a fix that we should be able to contribute back once we get it through some testing.  Should I reopen SAK-21398 and attach a patch or would it be more appropriate to open a new one?

Chris

From: <Maurer>, Christopher Maurer <chmaurer at iu.edu<mailto:chmaurer at iu.edu>>
Date: Friday, September 20, 2013 11:36 AM
To: Matthew Jones <matthew at longsight.com<mailto:matthew at longsight.com>>, Beth Kirschner <bkirschn at umich.edu<mailto:bkirschn at umich.edu>>
Cc: dev sakai <sakai-dev at collab.sakaiproject.org<mailto:sakai-dev at collab.sakaiproject.org>>

Subject: Re: [Building Sakai] Memory leak in Courier Service

We restart weekly on Sunday mornings around 6am.  But Sunday evening is some of our heaviest load, so by around 8pm, memory is about at capacity and GC starts thrashing essentially causing the servers to die a slow horrible death!
So restarts don't really help us in this situation.  And if it can build up in a little over 12 hours, that doesn't seem like a slow leak.  Though, we are running lots of sessions when we hit peak load!

Chris

From: Matthew Jones <matthew at longsight.com<mailto:matthew at longsight.com>>
Date: Friday, September 20, 2013 11:12 AM
To: Beth Kirschner <bkirschn at umich.edu<mailto:bkirschn at umich.edu>>
Cc: Christopher Maurer <chmaurer at iu.edu<mailto:chmaurer at iu.edu>>, dev sakai <sakai-dev at collab.sakaiproject.org<mailto:sakai-dev at collab.sakaiproject.org>>
Subject: Re: [Building Sakai] Memory leak in Courier Service

Chat (new or old) still uses it, new presence was the one that got switched from using it.

It's still an issue but wasn't a huge/fast leak and basically just meant the best practice was weekly restarts for highly active instances.


On Fri, Sep 20, 2013 at 11:07 AM, Beth Kirschner <bkirschn at umich.edu<mailto:bkirschn at umich.edu>> wrote:
We're still using the old chat, though I thought that the Courier Service was deprecated and so not used by the old chat?

- Beth

On Sep 20, 2013, at 10:00 AM, "Maurer, Christopher Wayne" <chmaurer at iu.edu<mailto:chmaurer at iu.edu>> wrote:

> https://jira.sakaiproject.org/browse/SAK-21398
>
> We believe we've been running into this old-ish issue lately.  We upgraded to 2.9.0 back in May but didn't notice issues until load really picked up at the beginning of the fall semester a couple weeks ago.  Are other 2.9 people still running the "old" style chat or has everyone switched over to the "neo" chat (I've been out of the sakai game for pretty much the whole year…is it still called that?)  Trying to determine if the best course of action is to try and get the old one working better, or switch to the new.  Thoughts?
>
> Chris
> _______________________________________________
> sakai-dev mailing list
> sakai-dev at collab.sakaiproject.org<mailto:sakai-dev at collab.sakaiproject.org>
> http://collab.sakaiproject.org/mailman/listinfo/sakai-dev
>
> TO UNSUBSCRIBE: send email to sakai-dev-unsubscribe at collab.sakaiproject.org<mailto:sakai-dev-unsubscribe at collab.sakaiproject.org> with a subject of "unsubscribe"

_______________________________________________
sakai-dev mailing list
sakai-dev at collab.sakaiproject.org<mailto:sakai-dev at collab.sakaiproject.org>
http://collab.sakaiproject.org/mailman/listinfo/sakai-dev

TO UNSUBSCRIBE: send email to sakai-dev-unsubscribe at collab.sakaiproject.org<mailto:sakai-dev-unsubscribe at collab.sakaiproject.org> with a subject of "unsubscribe"


_______________________________________________
sakai-dev mailing list
sakai-dev at collab.sakaiproject.org<mailto:sakai-dev at collab.sakaiproject.org>
http://collab.sakaiproject.org/mailman/listinfo/sakai-dev

TO UNSUBSCRIBE: send email to sakai-dev-unsubscribe at collab.sakaiproject.org<mailto:sakai-dev-unsubscribe at collab.sakaiproject.org> with a subject of "unsubscribe"

-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://collab.sakaiproject.org/pipermail/cle-release-team/attachments/20130927/0fad456a/attachment.html