[Building Sakai] Severe DB Performance Issues for 2.9

[Thomas, Gregory J]

A little bit of a followup to this.  We pulled the KNL-1011 change out of our build yesterday.  Not sure where in the code (yet!) it happens, but this was really pounding our database with tons of queries and also seems to lead to a scenario of causing a memory leak under really high load. (we sometimes have 5000-6000 users on the system at the same time) We had 3 different occasions where the system became unusable.  We're not sure yet if it's a combination of stuff from the patch, other code we have that might be unique to IU, or what, but we're definitely planning on investigating.

Since removing the code, our average response time has been anywhere from 77 - 120 ms for the past couple days.  With the patch, it was common for it average around 250 ms and still be stable before it grinded to a halt.

Greg
________________________________________
From: sakai-dev-bounces at collab.sakaiproject.org [sakai-dev-bounces at collab.sakaiproject.org] on behalf of Aaron Zeckoski [azeckoski at unicon.net]
Sent: Friday, September 06, 2013 1:35 PM
To: Matthew Jones
Cc: dev sakai
Subject: Re: [Building Sakai] Severe DB Performance Issues for 2.9

Looks like the majority of the change is moving the CacheInitializer
to utils and then checking for "memory."+cachename. Seems safe enough
to me. I'm thinking this should be merged to trunk for testing.

On Fri, Sep 6, 2013 at 1:21 PM, Matthew Jones <matthew at longsight.com> wrote:
> Hi Aaron,
>
> It looks like these caches needed to be configured in EhCacheFactoryBean.
> Possibly because of how they were wired?
>
> I attached a patch to a branch on
> https://jira.sakaiproject.org/browse/KNL-1118
>
> That allows the cache values to be set. It looks like all of the caches I've
> tried are working with this format now. Some review could be used. I don't
> have crucible setup for branches, but it's indexing now.
>
>
> On Wed, Sep 4, 2013 at 7:40 PM, Aaron Zeckoski <azeckoski at unicon.net> wrote:
>>
>> I think this can only work for caches registered with or created
>> through the memory service. Maybe the issue is how that cache is
>> created?
>> -AZ
>>
>>
>> On Wed, Sep 4, 2013 at 5:42 PM, Sam Ottenhoff <ottenhoff at longsight.com>
>> wrote:
>> >
>> >
>> >
>> > On Wed, Sep 4, 2013 at 5:39 PM, Lydia Li <lydial at stanford.edu> wrote:
>> >>
>> >> Sam/Matthew,
>> >>
>> >> I'm a bit confused about what you said.  I just tested in my dev
>> >> instance
>> >> running 2.9.x and this syntax still seems to work.
>> >>
>> >>
>> >>
>> >> memory.org.sakaiproject.site.impl.SiteCacheImpl.cache=timeToLiveSeconds=303,timeToIdleSeconds=346,maxElementsInMemory=10220
>> >
>> >
>> >
>> > Yes, that's the correct form, and it works perfectly for *that* cache.
>> > But
>> > it doesn't seem to work correctly on the SAKAI_SITE_PAGE_PROPERTY cache:
>> >
>> >   org.sakaiproject.db.BaseDbFlatStorage.SAKAI_SITE_PAGE_PROPERTY
>> >
>> >
>> > --Sam
>> >
>> > _______________________________________________
>> > sakai-dev mailing list
>> > sakai-dev at collab.sakaiproject.org
>> > http://collab.sakaiproject.org/mailman/listinfo/sakai-dev
>> >
>> > TO UNSUBSCRIBE: send email to
>> > sakai-dev-unsubscribe at collab.sakaiproject.org
>> > with a subject of "unsubscribe"
>>
>>
>>
>> --
>> Aaron Zeckoski - Software Architect - http://tinyurl.com/azprofile
>> _______________________________________________
>> sakai-dev mailing list
>> sakai-dev at collab.sakaiproject.org
>> http://collab.sakaiproject.org/mailman/listinfo/sakai-dev
>>
>> TO UNSUBSCRIBE: send email to
>> sakai-dev-unsubscribe at collab.sakaiproject.org with a subject of
>> "unsubscribe"
>
>



--
Aaron Zeckoski - Software Architect - http://tinyurl.com/azprofile
_______________________________________________
sakai-dev mailing list
sakai-dev at collab.sakaiproject.org
http://collab.sakaiproject.org/mailman/listinfo/sakai-dev

TO UNSUBSCRIBE: send email to sakai-dev-unsubscribe at collab.sakaiproject.org with a subject of "unsubscribe"