[Building Sakai] Sakai Memory Issues

Wed Oct 9 06:24:03 PDT 2013

John,

   Would your setup allow getting a heap dump early on before the node get
overloaded?  The server would be unavailable during the dump which may take
many minutes.  At Michigan we've gotten heap dumps by taking a server out
of the active cluster after running for a while, and then getting a heap
dump the next day when people with active sessions on that server have
drifted away.

- Dave

On Tue, Oct 8, 2013 at 12:10 PM, John Bush <jbush at anisakai.com> wrote:

> Ok this is interesting.  Jeremy, we tried to get a heap dump but our
> node was so over loaded at that point it was impossible.  The patch
> from IU is for a ConcurrentHashMap filling up.  I would have expected
> appdynamics to find that.
>
> I was going to try to use yourkit to look at the heap, I think it can
> do it, although I haven't used it for that purpose before.  Jeremy,
> it might be helpful if you could review this patch and dig in your
> heap and bit and see if it looks like that same thing.
>
>
> On Tue, Oct 8, 2013 at 8:51 AM, Thomas, Gregory J <gjthomas at iu.edu> wrote:
> > Indiana recently had a memory leak issue with the delivery objects in
> > courier and the chat tool.  We have a patch here:
> > https://jira.sakaiproject.org/browse/SAK-21398  It hasn't been committed
> > to trunk yet as there were some changes that were preferred to be done
> > first on its implementation.  However, we have the current patch in our
> > production code as of Thursday last week.  It passed our load testing and
> > it passed our dreaded Sunday evening for high server load/usage.
> >
> > Not sure if it's related to issues brought up here, but thought I'd throw
> > it out there as something we've definitely ran into.
> >
> > Thanks,
> > Greg
> >
> >
> >
> > On 10/8/13 10:57 AM, "Kusnetz, Jeremy" <JKusnetz at APUS.EDU> wrote:
> >
> >>We have been seeing similar memory problems since upgrading to 2.9.3
> >>also.  The oldgen space just continually grows until it's out of space.
> >>GC just doesn't seem to clean everything up.  We have tried lots of
> >>different combinations of GC, the best we can do is slow down how quickly
> >>we run out of memory so we are forced to have to restart Sakai multiple
> >>times a week.
> >>
> >>Interestingly we are running two instances of Sakai all running from the
> >>same code base.  Once instance has the memory issue while the other does
> >>not.
> >>
> >>The one that does NOT have the memory issues is primarily using jforums,
> >>assignment1, and gradebook1.  The one that IS having memory issues is
> >>primarily using msgcntr, assignment2, and gradebook2.  There is a small
> >>amount of jforum usage in the one with memory issues, but it's only in a
> >>couple of sites.
> >>
> >>There are a few other differences.  The one with memory issues has 35
> >>sakai nodes, while the one without memory issues is only 2 sakai nodes.
> >>But on average the one with 2 nodes has more sessions on each of those
> >>nodes then the one with 35 nodes.  The one without memory issues is using
> >>LDAP, while the one without memory issues is not using LDAP, all users
> >>are local but we do use the soap axis SakaiPortalLogin for most of it's
> >>session creation.
> >>
> >>We are also using appdynamics to try to find the memory leaks, but it's
> >>automatic leak detection only tracks Map and Collection libraries.  There
> >>is another tool to map any custom object, but we would need to know what
> >>object to look for.  The only thing appdynamics seems to think are memory
> >>leaks with it's automatic leak detection are the various ehcaches.
> >>
> >>We did try dumping the heap and opening it in the Eclipse Memory
> >>Analyzer.  I don't have much experience with it.  Even though the heap
> >>dump was close to 7GB, it only identified about 2.5GB, the rest were
> >>unreferenced objects.  I'm not sure why those objects weren't GCed.
> >>
> >>
> >>
> >>-----Original Message-----
> >>From: sakai-dev-bounces at collab.sakaiproject.org
> >>[mailto:sakai-dev-bounces at collab.sakaiproject.org] On Behalf Of John
> Bush
> >>Sent: Tuesday, October 08, 2013 9:54 AM
> >>To: Mike Jennings
> >>Cc: sakai-dev Developers
> >>Subject: Re: [Building Sakai] Sakai Memory Issues
> >>
> >>we are on the 2.9 tag.   I don't have a smokin' gun yet to really say
> >>if that's the root cause or not.  We are using appdynamics to look at
> >>things in production.  What we are seeing is a single request run this
> >>type of query literary hundreds of times.
> >>
> >>[SELECT special_access_id, forum_id, topic_id, start_date,
> >>hide_until_open, end_date, allow_until_date, override_start_date,
> >>override_hide_until_open, override_end_date, override_allow_until_date,
> >>password, lock_end_date, users FROM jforum_special_access WHERE forum_id
> >>= '116964'  AND topic_id > 0]
> >>
> >>Each one is fast and our database has no issue.  Eventually we see
> >>getting connections taking awhile and threads get blocked but we aren't
> >>running into any type of bottleneck load or connection wise on the db.  I
> >>think the appservers are simply running stupid amounts of queries and
> >>just eating up connections and then blocking and keeping things in
> >>memory.  Its just a hunch right now.  Its the best we've got so far.
> >>
> >>I wrote some code to cache this stuff yesterday, we are QA'ing today.
> >>Again, I'm not sure its jforum, but jforum certainly isn't helping.
> >>
> >>On Tue, Oct 8, 2013 at 6:22 AM, Mike Jennings <mike_jennings at unc.edu>
> >>wrote:
> >>> John,
> >>>
> >>> We are using JForum and Sakai 2.9.2 and are not seeing any Memory
> >>> Issues to date.  What version of JForum's do you suspect this memory
> >>>leak in?
> >>>
> >>> Mike Jennings
> >>>
> >>>
> >>> On 10/07/2013 05:39 PM, John Bush wrote:
> >>>>
> >>>> We've been having an issue like this with one client, we think it
> >>>> might be related to jforum.  Do you use jforum ?
> >>>>
> >>>> On Mon, Oct 7, 2013 at 1:11 PM, William Karavites
> >>>> <willkara at oit.rutgers.edu> wrote:
> >>>>>
> >>>>> Hello Community,
> >>>>>
> >>>>> I was wondering if there were any outstanding or recently fixed
> >>>>> memory usage issues for Sakai. We recently came across one that took
> >>>>> Sakai down twice in a night for 30 minutes each. We found the
> >>>>> culprit for that one and want to try and be proactive for any others
> >>>>> in the future. We are currently running Sakai 2.9.1 and Kernel
> >>>>> 1.3.1.
> >>>>>
> >>>>>
> >>>>> So if there are any improved memory usage patches in Sakai, please
> >>>>> let me know.
> >>>>>
> >>>>>
> >>>>> Thank you,
> >>>>> William Karavites
> >>>>>
> >>>>> ------------------------------------
> >>>>> William Karavites
> >>>>> Application Programmer
> >>>>> OIT/OIRT- Rutgers University
> >>>>> Office: 732-445-8726
> >>>>> Cell: 732-822-9405
> >>>>> willkara at rutgers.edu
> >>>>> ------------------------------------
> >>>>>
> >>>>>
> >>>>> _______________________________________________
> >>>>> sakai-dev mailing list
> >>>>> sakai-dev at collab.sakaiproject.org
> >>>>> http://collab.sakaiproject.org/mailman/listinfo/sakai-dev
> >>>>>
> >>>>> TO UNSUBSCRIBE: send email to
> >>>>> sakai-dev-unsubscribe at collab.sakaiproject.org
> >>>>> with a subject of "unsubscribe"
> >>>>
> >>>>
> >>>>
> >>>>
> >>>
> >>> --
> >>> =========================================
> >>> Mike Jennings
> >>> Teaching and Learning Developer
> >>> T: 919.843.5013
> >>> E: mike_jennings at unc.edu
> >>
> >>
> >>
> >>--
> >>John Bush
> >>602-490-0470
> >>
> >>** This message is neither private nor confidential in fact the US
> >>government is storing it in a warehouse located in Utah for future data
> >>mining use cases should they arise. **
> >>_______________________________________________
> >>sakai-dev mailing list
> >>sakai-dev at collab.sakaiproject.org
> >>http://collab.sakaiproject.org/mailman/listinfo/sakai-dev
> >>
> >>TO UNSUBSCRIBE: send email to
> >>sakai-dev-unsubscribe at collab.sakaiproject.org with a subject of
> >>"unsubscribe"
> >>This message is private and confidential. If you have received it in
> >>error, please notify the sender and remove it from your system.
> >>
> >>_______________________________________________
> >>sakai-dev mailing list
> >>sakai-dev at collab.sakaiproject.org
> >>http://collab.sakaiproject.org/mailman/listinfo/sakai-dev
> >>
> >>TO UNSUBSCRIBE: send email to
> >>sakai-dev-unsubscribe at collab.sakaiproject.org with a subject of
> >>"unsubscribe"
> >
>
>
>
> --
> John Bush
> 602-490-0470
>
> ** This message is neither private nor confidential in fact the US
> government is storing it in a warehouse located in Utah for future
> data mining use cases should they arise. **
> _______________________________________________
> sakai-dev mailing list
> sakai-dev at collab.sakaiproject.org
> http://collab.sakaiproject.org/mailman/listinfo/sakai-dev
>
> TO UNSUBSCRIBE: send email to
> sakai-dev-unsubscribe at collab.sakaiproject.org with a subject of
> "unsubscribe"
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://collab.sakaiproject.org/pipermail/sakai-dev/attachments/20131009/e4d57b1c/attachment.html 