[sakai2-tcc] KNL-652 site caching testing (UMich preliminary testing)

Tue Mar 15 05:45:21 PDT 2011

OK

KNL-293 - already in the 1.2 kenel
KNL-654 - ditto
KNL-662 - ditto
KNL-664 - ditto

These match what we're running in production at UCT

D


On 03/15/2011 02:38 PM, David Haines wrote:
> There are 4 caching changes that should be added as a group.  When we added just the site caching fix other issues popped up that quickly led to similar problems.  We needed KNL-293, KNL-654, KNL-662, and KNL-664 to get memory under control.  
>
> The problem clearly doesn't make Sakai useless since the problem exists since at least 2.6 and some people have been running that code for a long time.  I wouldn't run or recommend running 2.7 or 2.8 in production without these fixes but, in fact, it is likely that many installations aren't big enough for this to make a problem in the short term.  Does it make any sense to release a 2.8.0 "preview" on time and then a final with these fixes? 2.8 as it stands should be useful for preparing an installation of 2.8.
>
> IMHO it is not a good idea for Sakai to release code with a serious problem, but that is a political not a technical opinion.
>
> - Dave
>
>
> David Haines
> CTools Developer
> Digital Media Commons
> University of Michigan 
> dlhaines at umich.edu
>
>
>
>
> On Mar 15, 2011, at 5:32 AM, Anthony Whyte wrote:
>
>   
>> It's a blocker against kernel 1.3.0 (sakai-2.8 binds to kernel 1.2).  For sakai-2.8.0 it is being tracked as a known issue and David Horwitz and I have held back from including the fix in kernel 1.2/1.1/1.0 releases while we await the results of UMich testing.
>>
>> I'd like to hear from the UMich members of the TCC list regarding their assessment of the site caching fix (r88169-70) and the implications for sakai-2.8.0 if it is included/excluded.
>>
>> Cheers,
>>
>> Anthony   
>>
>>
>> On Mar 14, 2011, at 11:21 PM, May, Megan Marie wrote:
>>
>>     
>>> Isn't this already on the blocker list for 2.8 or are you proposing an elevation of the bug?
>>>
>>> Megan
>>>
>>> -----Original Message-----
>>> From: sakai2-tcc-bounces at collab.sakaiproject.org [mailto:sakai2-tcc-bounces at collab.sakaiproject.org] On Behalf Of Aaron Zeckoski
>>> Sent: Friday, March 11, 2011 8:55 AM
>>> To: Anthony Whyte
>>> Cc: sakai2-tcc Committee
>>> Subject: Re: [sakai2-tcc] KNL-652 site caching testing (UMich preliminary testing)
>>>
>>> I think that is a really good plan.
>>> It may be worth backporting to 2.7 as well.
>>> -AZ
>>>
>>>
>>> On Fri, Mar 11, 2011 at 8:38 AM, Anthony Whyte <arwhyte at umich.edu> wrote:
>>>       
>>>> I recommend that if Michigan's production testing continues to proves 
>>>> positive we should include this fix in sakai-2.8.0, even if it means 
>>>> we delay the release a further 7-10 days in order to work it in.
>>>> Cheers,
>>>> Anthony
>>>>
>>>> Begin forwarded message:
>>>>
>>>> From: "David Haines (JIRA)" <bugs-admin at sakaiproject.org>
>>>> Date: March 11, 2011 3:25:42 PM GMT+02:00
>>>> To: mt-jira at collab.sakaiproject.org
>>>> Subject: [mt-jira] [Sakai Jira] Commented: (KNL-652) SIte caching is 
>>>> broken in Kernel 1.1.9
>>>>
>>>>
>>>>   [
>>>> https://jira.sakaiproject.org/browse/KNL-652?page=com.atlassian.jira.p
>>>> lugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=122629#c
>>>> omment-122629
>>>> ]
>>>>
>>>> David Haines commented on KNL-652:
>>>> ----------------------------------
>>>>
>>>> Load testing at Michigan using a lightly patched 1.1.9 kernel 
>>>> indicates that the memory leak is significantly reduced.  This 
>>>> required applying 4 patches related to caching: KNL-293, KNL-654, 
>>>> KNL-662, and KNL-664 to fix multiple leaks.  Our tests indicate that a 
>>>> site cache size of 10000 gives a site cache hit rate of around 90%.  
>>>> We should have a week or more of production experience with these 
>>>> changes, applied to a 1.1.10 kernel, as of the week of 4/4/11.
>>>>
>>>> SIte caching is broken in Kernel 1.1.9
>>>>
>>>> --------------------------------------
>>>>
>>>>               Key: KNL-652
>>>>
>>>>               URL: https://jira.sakaiproject.org/browse/KNL-652
>>>>
>>>>           Project: Kernel - K1
>>>>
>>>>        Issue Type: Bug
>>>>
>>>>        Components: Impl
>>>>
>>>>  Affects Versions: 1.1.9
>>>>
>>>>          Reporter: David Haines
>>>>
>>>>          Assignee: David Horwitz
>>>>
>>>>          Priority: Blocker
>>>>
>>>>           Fix For: 1.3.0
>>>>
>>>>       Attachments: Classes in Heap.png, Top Consumers.png
>>>>
>>>>
>>>> Site caching is broken in the K1 kernel 1.1.9 release.  Sites objects 
>>>> themselves are cached but when sites are evicted from the site cache 
>>>> the secondary, in memory, caches for tools, pages, and groups are not 
>>>> cleaned up, the references to the objects remain and those objects can 
>>>> not be garbage collected.  Over time this leads to large numbers of 
>>>> objects that won't be used but will consume memory.  At Michigan, 
>>>> after not restarting Sakai for a couple of weeks we ended up with 
>>>> almost 2GB of memory devoted to the site cache.  That caused long 
>>>> periods of fruitless garbage collection and a degradation of service.
>>>>
>>>> The problem is avoided in the short run by restarting Sakai instances 
>>>> or by manually clearing the caches with the Admin memory tool.
>>>>
>>>> A patch to address this is being tested at Michigan.
>>>>
>>>> Note that the only way to control Ehcache cache memory consumption for 
>>>> a memory only cache is to limit the number of elements in the cache.  
>>>> KNL-293 should also be applied when the site caching is fixed so that 
>>>> there is a way to adjust the site cache size to match local requirements.
>>>>
>>>> --
>>>> This message is automatically generated by JIRA.
>>>> -
>>>> For more information on JIRA, see: 
>>>> http://www.atlassian.com/software/jira
>>>>
>>>>
>>>> _______________________________________________
>>>> mt-jira mailing list
>>>> mt-jira at collab.sakaiproject.org
>>>> http://collab.sakaiproject.org/mailman/listinfo/mt-jira
>>>>
>>>>
>>>>
>>>>
>>>> _______________________________________________
>>>> sakai2-tcc mailing list
>>>> sakai2-tcc at collab.sakaiproject.org
>>>> http://collab.sakaiproject.org/mailman/listinfo/sakai2-tcc
>>>>
>>>>
>>>>         
>>>
>>>
>>> --
>>> Aaron Zeckoski - Software Engineer - http://tinyurl.com/azprofile _______________________________________________
>>> sakai2-tcc mailing list
>>> sakai2-tcc at collab.sakaiproject.org
>>> http://collab.sakaiproject.org/mailman/listinfo/sakai2-tcc
>>>
>>>
>>>       
>> _______________________________________________
>> sakai2-tcc mailing list
>> sakai2-tcc at collab.sakaiproject.org
>> http://collab.sakaiproject.org/mailman/listinfo/sakai2-tcc
>>     
> _______________________________________________
> sakai2-tcc mailing list
> sakai2-tcc at collab.sakaiproject.org
> http://collab.sakaiproject.org/mailman/listinfo/sakai2-tcc
>