[sakai2-tcc] KNL-652 site caching testing (UMich preliminary testing)

David Haines dlhaines at umich.edu
Tue Mar 15 05:38:45 PDT 2011 

There are 4 caching changes that should be added as a group.  When we added just the site caching fix other issues popped up that quickly led to similar problems.  We needed KNL-293, KNL-654, KNL-662, and KNL-664 to get memory under control.  

The problem clearly doesn't make Sakai useless since the problem exists since at least 2.6 and some people have been running that code for a long time.  I wouldn't run or recommend running 2.7 or 2.8 in production without these fixes but, in fact, it is likely that many installations aren't big enough for this to make a problem in the short term.  Does it make any sense to release a 2.8.0 "preview" on time and then a final with these fixes? 2.8 as it stands should be useful for preparing an installation of 2.8.

IMHO it is not a good idea for Sakai to release code with a serious problem, but that is a political not a technical opinion.

- Dave


David Haines
CTools Developer
Digital Media Commons
University of Michigan 
dlhaines at umich.edu




On Mar 15, 2011, at 5:32 AM, Anthony Whyte wrote:

> It's a blocker against kernel 1.3.0 (sakai-2.8 binds to kernel 1.2).  For sakai-2.8.0 it is being tracked as a known issue and David Horwitz and I have held back from including the fix in kernel 1.2/1.1/1.0 releases while we await the results of UMich testing.
> 
> I'd like to hear from the UMich members of the TCC list regarding their assessment of the site caching fix (r88169-70) and the implications for sakai-2.8.0 if it is included/excluded.
> 
> Cheers,
> 
> Anthony   
> 
> 
> On Mar 14, 2011, at 11:21 PM, May, Megan Marie wrote:
> 
>> Isn't this already on the blocker list for 2.8 or are you proposing an elevation of the bug?
>> 
>> Megan
>> 
>> -----Original Message-----
>> From: sakai2-tcc-bounces at collab.sakaiproject.org [mailto:sakai2-tcc-bounces at collab.sakaiproject.org] On Behalf Of Aaron Zeckoski
>> Sent: Friday, March 11, 2011 8:55 AM
>> To: Anthony Whyte
>> Cc: sakai2-tcc Committee
>> Subject: Re: [sakai2-tcc] KNL-652 site caching testing (UMich preliminary testing)
>> 
>> I think that is a really good plan.
>> It may be worth backporting to 2.7 as well.
>> -AZ
>> 
>> 
>> On Fri, Mar 11, 2011 at 8:38 AM, Anthony Whyte <arwhyte at umich.edu> wrote:
>>> I recommend that if Michigan's production testing continues to proves 
>>> positive we should include this fix in sakai-2.8.0, even if it means 
>>> we delay the release a further 7-10 days in order to work it in.
>>> Cheers,
>>> Anthony
>>> 
>>> Begin forwarded message:
>>> 
>>> From: "David Haines (JIRA)" <bugs-admin at sakaiproject.org>
>>> Date: March 11, 2011 3:25:42 PM GMT+02:00
>>> To: mt-jira at collab.sakaiproject.org
>>> Subject: [mt-jira] [Sakai Jira] Commented: (KNL-652) SIte caching is 
>>> broken in Kernel 1.1.9
>>> 
>>> 
>>>   [
>>> https://jira.sakaiproject.org/browse/KNL-652?page=com.atlassian.jira.p
>>> lugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=122629#c
>>> omment-122629
>>> ]
>>> 
>>> David Haines commented on KNL-652:
>>> ----------------------------------
>>> 
>>> Load testing at Michigan using a lightly patched 1.1.9 kernel 
>>> indicates that the memory leak is significantly reduced.  This 
>>> required applying 4 patches related to caching: KNL-293, KNL-654, 
>>> KNL-662, and KNL-664 to fix multiple leaks.  Our tests indicate that a 
>>> site cache size of 10000 gives a site cache hit rate of around 90%.  
>>> We should have a week or more of production experience with these 
>>> changes, applied to a 1.1.10 kernel, as of the week of 4/4/11.
>>> 
>>> SIte caching is broken in Kernel 1.1.9
>>> 
>>> --------------------------------------
>>> 
>>>               Key: KNL-652
>>> 
>>>               URL: https://jira.sakaiproject.org/browse/KNL-652
>>> 
>>>           Project: Kernel - K1
>>> 
>>>        Issue Type: Bug
>>> 
>>>        Components: Impl
>>> 
>>>  Affects Versions: 1.1.9
>>> 
>>>          Reporter: David Haines
>>> 
>>>          Assignee: David Horwitz
>>> 
>>>          Priority: Blocker
>>> 
>>>           Fix For: 1.3.0
>>> 
>>>       Attachments: Classes in Heap.png, Top Consumers.png
>>> 
>>> 
>>> Site caching is broken in the K1 kernel 1.1.9 release.  Sites objects 
>>> themselves are cached but when sites are evicted from the site cache 
>>> the secondary, in memory, caches for tools, pages, and groups are not 
>>> cleaned up, the references to the objects remain and those objects can 
>>> not be garbage collected.  Over time this leads to large numbers of 
>>> objects that won't be used but will consume memory.  At Michigan, 
>>> after not restarting Sakai for a couple of weeks we ended up with 
>>> almost 2GB of memory devoted to the site cache.  That caused long 
>>> periods of fruitless garbage collection and a degradation of service.
>>> 
>>> The problem is avoided in the short run by restarting Sakai instances 
>>> or by manually clearing the caches with the Admin memory tool.
>>> 
>>> A patch to address this is being tested at Michigan.
>>> 
>>> Note that the only way to control Ehcache cache memory consumption for 
>>> a memory only cache is to limit the number of elements in the cache.  
>>> KNL-293 should also be applied when the site caching is fixed so that 
>>> there is a way to adjust the site cache size to match local requirements.
>>> 
>>> --
>>> This message is automatically generated by JIRA.
>>> -
>>> For more information on JIRA, see: 
>>> http://www.atlassian.com/software/jira
>>> 
>>> 
>>> _______________________________________________
>>> mt-jira mailing list
>>> mt-jira at collab.sakaiproject.org
>>> http://collab.sakaiproject.org/mailman/listinfo/mt-jira
>>> 
>>> 
>>> 
>>> 
>>> _______________________________________________
>>> sakai2-tcc mailing list
>>> sakai2-tcc at collab.sakaiproject.org
>>> http://collab.sakaiproject.org/mailman/listinfo/sakai2-tcc
>>> 
>>> 
>> 
>> 
>> 
>> --
>> Aaron Zeckoski - Software Engineer - http://tinyurl.com/azprofile _______________________________________________
>> sakai2-tcc mailing list
>> sakai2-tcc at collab.sakaiproject.org
>> http://collab.sakaiproject.org/mailman/listinfo/sakai2-tcc
>> 
>> 
> 
> _______________________________________________
> sakai2-tcc mailing list
> sakai2-tcc at collab.sakaiproject.org
> http://collab.sakaiproject.org/mailman/listinfo/sakai2-tcc

More information about the sakai2-tcc mailing list