[sakai2-tcc] KNL-652 site caching testing (UMich preliminary testing)
David Horwitz
david.horwitz at uct.ac.za
Wed Mar 16 02:55:42 PDT 2011
Hi All,
As opinion seems to be shifting and so as not to hold up the release
unnecessarily before the start of the US day. I have:
1) Merged KNL-652 to 1.2.x
2) released kernel 1.2.0-rc13
3) brought up qa1-za with this latest kernel
All this is easily reversible if it doesn't meet with the groups approval.
D
On 03/15/2011 07:45 PM, Noah Botimer wrote:
> I think these are some of the most carefully and expertly executed changes in, or considered for, the 2.8 release. I've reviewed them and am comfortable with them. UCT is running them. We are load testing them at scale. They've been tracked with profiling and heap analysis over repeated tests...
>
> Almost none of our code base has had this level of diligence applied before landing in trunk. A bug is fixed with the same functionality, better code -- and no tradeoffs.
>
> I say ship it.
>
> I have no reservations about including them. I'm also not willing to lobby strongly. Both sides have their virtues and I've spoken my preference.
>
> I'm much more concerned about the hundreds of commits already in that haven't seen much review -- let alone feature/change documentation. There were many that touched dozens or hundreds of files and it's just very difficult to verify changes in increments of that size. This one doesn't scare me a bit.
>
> Thanks,
> -Noah
>
> On Mar 15, 2011, at 5:32 AM, Anthony Whyte wrote:
>
>
>> It's a blocker against kernel 1.3.0 (sakai-2.8 binds to kernel 1.2). For sakai-2.8.0 it is being tracked as a known issue and David Horwitz and I have held back from including the fix in kernel 1.2/1.1/1.0 releases while we await the results of UMich testing.
>>
>> I'd like to hear from the UMich members of the TCC list regarding their assessment of the site caching fix (r88169-70) and the implications for sakai-2.8.0 if it is included/excluded.
>>
>> Cheers,
>>
>> Anthony
>>
>>
>> On Mar 14, 2011, at 11:21 PM, May, Megan Marie wrote:
>>
>>
>>> Isn't this already on the blocker list for 2.8 or are you proposing an elevation of the bug?
>>>
>>> Megan
>>>
>>> -----Original Message-----
>>> From: sakai2-tcc-bounces at collab.sakaiproject.org [mailto:sakai2-tcc-bounces at collab.sakaiproject.org] On Behalf Of Aaron Zeckoski
>>> Sent: Friday, March 11, 2011 8:55 AM
>>> To: Anthony Whyte
>>> Cc: sakai2-tcc Committee
>>> Subject: Re: [sakai2-tcc] KNL-652 site caching testing (UMich preliminary testing)
>>>
>>> I think that is a really good plan.
>>> It may be worth backporting to 2.7 as well.
>>> -AZ
>>>
>>>
>>> On Fri, Mar 11, 2011 at 8:38 AM, Anthony Whyte <arwhyte at umich.edu> wrote:
>>>
>>>> I recommend that if Michigan's production testing continues to proves
>>>> positive we should include this fix in sakai-2.8.0, even if it means
>>>> we delay the release a further 7-10 days in order to work it in.
>>>> Cheers,
>>>> Anthony
>>>>
>>>> Begin forwarded message:
>>>>
>>>> From: "David Haines (JIRA)" <bugs-admin at sakaiproject.org>
>>>> Date: March 11, 2011 3:25:42 PM GMT+02:00
>>>> To: mt-jira at collab.sakaiproject.org
>>>> Subject: [mt-jira] [Sakai Jira] Commented: (KNL-652) SIte caching is
>>>> broken in Kernel 1.1.9
>>>>
>>>>
>>>> [
>>>> https://jira.sakaiproject.org/browse/KNL-652?page=com.atlassian.jira.p
>>>> lugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=122629#c
>>>> omment-122629
>>>> ]
>>>>
>>>> David Haines commented on KNL-652:
>>>> ----------------------------------
>>>>
>>>> Load testing at Michigan using a lightly patched 1.1.9 kernel
>>>> indicates that the memory leak is significantly reduced. This
>>>> required applying 4 patches related to caching: KNL-293, KNL-654,
>>>> KNL-662, and KNL-664 to fix multiple leaks. Our tests indicate that a
>>>> site cache size of 10000 gives a site cache hit rate of around 90%.
>>>> We should have a week or more of production experience with these
>>>> changes, applied to a 1.1.10 kernel, as of the week of 4/4/11.
>>>>
>>>> SIte caching is broken in Kernel 1.1.9
>>>>
>>>> --------------------------------------
>>>>
>>>> Key: KNL-652
>>>>
>>>> URL: https://jira.sakaiproject.org/browse/KNL-652
>>>>
>>>> Project: Kernel - K1
>>>>
>>>> Issue Type: Bug
>>>>
>>>> Components: Impl
>>>>
>>>> Affects Versions: 1.1.9
>>>>
>>>> Reporter: David Haines
>>>>
>>>> Assignee: David Horwitz
>>>>
>>>> Priority: Blocker
>>>>
>>>> Fix For: 1.3.0
>>>>
>>>> Attachments: Classes in Heap.png, Top Consumers.png
>>>>
>>>>
>>>> Site caching is broken in the K1 kernel 1.1.9 release. Sites objects
>>>> themselves are cached but when sites are evicted from the site cache
>>>> the secondary, in memory, caches for tools, pages, and groups are not
>>>> cleaned up, the references to the objects remain and those objects can
>>>> not be garbage collected. Over time this leads to large numbers of
>>>> objects that won't be used but will consume memory. At Michigan,
>>>> after not restarting Sakai for a couple of weeks we ended up with
>>>> almost 2GB of memory devoted to the site cache. That caused long
>>>> periods of fruitless garbage collection and a degradation of service.
>>>>
>>>> The problem is avoided in the short run by restarting Sakai instances
>>>> or by manually clearing the caches with the Admin memory tool.
>>>>
>>>> A patch to address this is being tested at Michigan.
>>>>
>>>> Note that the only way to control Ehcache cache memory consumption for
>>>> a memory only cache is to limit the number of elements in the cache.
>>>> KNL-293 should also be applied when the site caching is fixed so that
>>>> there is a way to adjust the site cache size to match local requirements.
>>>>
>>>> --
>>>> This message is automatically generated by JIRA.
>>>> -
>>>> For more information on JIRA, see:
>>>> http://www.atlassian.com/software/jira
>>>>
>>>>
>>>> _______________________________________________
>>>> mt-jira mailing list
>>>> mt-jira at collab.sakaiproject.org
>>>> http://collab.sakaiproject.org/mailman/listinfo/mt-jira
>>>>
>>>>
>>>>
>>>>
>>>> _______________________________________________
>>>> sakai2-tcc mailing list
>>>> sakai2-tcc at collab.sakaiproject.org
>>>> http://collab.sakaiproject.org/mailman/listinfo/sakai2-tcc
>>>>
>>>>
>>>>
>>>
>>>
>>> --
>>> Aaron Zeckoski - Software Engineer - http://tinyurl.com/azprofile _______________________________________________
>>> sakai2-tcc mailing list
>>> sakai2-tcc at collab.sakaiproject.org
>>> http://collab.sakaiproject.org/mailman/listinfo/sakai2-tcc
>>>
>>>
>>>
>> _______________________________________________
>> sakai2-tcc mailing list
>> sakai2-tcc at collab.sakaiproject.org
>> http://collab.sakaiproject.org/mailman/listinfo/sakai2-tcc
>>
> _______________________________________________
> sakai2-tcc mailing list
> sakai2-tcc at collab.sakaiproject.org
> http://collab.sakaiproject.org/mailman/listinfo/sakai2-tcc
>
More information about the sakai2-tcc
mailing list