[Deploying Sakai] 2.7 load

Fri Sep 3 16:26:52 PDT 2010

It appears that we are running with KNL-259. we're getting reasonable caching of SAKAI_SITE_PAGE_PROPERTY. (I'm not quite sure how that patch got into our image, although we do check for patches that aren't in the release that look important.)

Recall that we're also seeing high loads on the front ends, though not be as much. I wondered if somehow offered load is higher, but we are not seeing higher than normal traffic in webalizer reports.  While I think our DB performance would be improved by caching the SITE_TOOL tables, I have a feeling there's something else going on. I can't help wondering if we're doing something local, though I can't quite imagine what we could be doing that would have this effect. Rutgers-specific tables don't show very high on the list. I also tried increasing the presence timeout by a factor of 3, and it had almost no effect.

I agree that it's not an emergency. It's just worrisome. It could even be something we've done. We're about to increase the CPU on the front ends significantly. We can probably increase it a bit for the database server if we have to, though we're already using a fairly recent 8-core system.

My sense is that caching the TOOL tables would get us into reasonable shape. If nothing else shows up I'll probably do that, though I'd rather have someone on the kernel team do it.

On Sep 3, 2010, at 8:52:30 AM, csev wrote:

> But lets be really clear here - lots of folks are running 2.6 in production with decent performance with KNL-259 in place and no further tweaks.  KNL-259 was identified and fixed very early in the 2.6 process.
> 
> The kernel team did not put the patch into trunk - so for 2.7 we had a performance regression.  The fix is to use the code in KNL-259 and hopefully put it into 2-6-x, 2-7-x and trunk if that has not already been done.
> 
> What you are proposing in KNL-577 is new work and further performance improvement above and beyond KNL-259 which probably would be a good idea.  Lets just make sure that this work you are doing is not considered part of the "lets fix the 2.7 problem" and part of overall tuning.
> 
> That suggests careful testing of your new work before pushing it out since there is no "emergency" or "rush" associated with KNL-577.  Getting KNL-259 into 2.7 (and 2.6 if not already done) production is something we need to get done quickly as it is well tested with lots of production experience for well over a year.
> 
> /Chuck
> 
> On Sep 3, 2010, at 3:44 AM, David Horwitz wrote:
> 
>> Looking at the code what seems to have happened is that Chuck added caching to BaseDbFlat storage that caches all properties collections accesses that way. The properties in question are accessed directly from the Db so are not cached. I'll have a look at adding some caching here - it should be pretty straight foward. http://jira.sakaiproject.org/browse/KNL-577
>> 
>> D
>> 
>> On 09/02/2010 08:47 PM, Matthew Jones wrote:
>>> 
>>> Well, the SITE_PAGE_PROPERTY should be fixed by the issue KNL-259 that Noah/Zhen worked on. It didn't get committed to the main sakai repo for inclusing until 2.7.1/2.6.3, I'm betting this will help out a little. This was run both at IU and Michigan and resulted both in decreased queries and increased caching of these properties.
>>> 
>>> Not sure about that TOOL_PROPERTY one but it "seems" like that one should/could be cached as well. I don't see one specifically set up for it though in the bean in db-components.xml or displaying in the memory tool.
>>> 
>>> (Sample cache report)
>>> org.sakaiproject.db.BaseDbFlatStorage.SAKAI_ALIAS_PROPERTY: count:0 hits:34 misses:34 hit%:50
>>> org.sakaiproject.db.BaseDbFlatStorage.SAKAI_REALM_PROPERTY: count:85 hits:180440 misses:59292 hit%:75
>>> org.sakaiproject.db.BaseDbFlatStorage.SAKAI_SITE_GROUP_PROPERTY: count:0 hits:0 misses:24474 hit%:0
>>> org.sakaiproject.db.BaseDbFlatStorage.SAKAI_SITE_PAGE_PROPERTY: count:374 hits:12941 misses:17298 hit%:42
>>> org.sakaiproject.db.BaseDbFlatStorage.SAKAI_SITE_PROPERTY: count:329 hits:725659 misses:187371 hit%:79
>>> org.sakaiproject.db.BaseDbFlatStorage.SAKAI_USER_PROPERTY: count:0 hits:12 misses:12 hit%:50
>>> 
>>> Stephen mentioned back in this JIRA [1] he also didn't notice this SAKAI_SITE_TOOL_PROPERTY show up in the cache list either. Would be worth investigating if this is possible, why it wasn't done.
>>> 
>>> In talking to Chuck S, he recommended that kernel property (and other) caching be completely reviewed and improved, but this sounds like a big (2.9, 2.10) job.
>>> 
>>> In Glenn's only blog post (that I know of) [2], he talked about improving the caching and performance of all of those SAKAI_REALM functions. Those hit us pretty hard too (with realm queries and presence driving most of the database traffic). Especially on some sites with lots of users and resources in content.
>>> 
>>> [1] http://jira.sakaiproject.org/browse/SAK-13886?focusedCommentId=65873&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#action_65873
>>> 
>>> [2] http://galvanometer.blogspot.com/
>>> 
>>> On Thu, Sep 2, 2010 at 8:47 AM, Hedrick Charles <hedrick at rutgers.edu> wrote:
>>> The students just got back for the Fall, so this is the first time we've run 2.7 under a realistic load. It looks to me like the CPU usage on both front ends and database is significantly higher, by at least a factor of 2. This is probably not going to kill us, but it worries me.
>>> 
>>> I took a very brief snapshot of database queries during the day yesterday. Here's the tables it was querying. The scripts were pretty simple-minded, so this isn't perfect. But it should give an indication. The top queries seem to be simple lookups on single tables, so caching should work very well. I see no evidence that this data is actually being cached.
>>> 
>>> 
>>>   2561 SAKAI_SITE_TOOL_PROPERTY
>>>   2196 SAKAI_SITE_TOOL
>>>   1237 SAKAI_REALM_RL_FN,SAKAI_REALM
>>>   1053 SAKAI_SITE_PAGE_PROPERTY
>>>    253 SAKAI_SITE_PAGE
>>>    237
>>>    181 SAKAI_REALM_ROLE
>>>    108 SAKAI_PRIVACY_RECORD
>>>     89 SAKAI_SITE,SAKAI_SITE_USER
>>>     84 CONTENT_COLLECTION
>>>     70 SAKAI_SITE_PROPERTY
>>>     64 SAKAI_SITE
>>>     63 CONTENT_RESOURCE
>>>     48 SAKAI_REALM_ROLE_DESC
>>>     48 SAKAI_REALM_RL_GR
>>>     48 SAKAI_REALM_RL_FN
>>>     47 rutgers_user
>>>     33 SAKAI_ALIAS
>>>     30 SAKAI_SYLLABUS_ITEM
>>>     24 SAKAI_SESSION
>>>     24 SAKAI_REALM_FUNCTION
>>>     22 SAKAI_SITE_GROUP_PROPERTY
>>>     22 SAKAI_SITE_GROUP
>>>     20 ANNOUNCEMENT_CHANNEL
>>>     19 ANNOUNCEMENT_MESSAGE
>>>     15 SST_EVENTS
>>>     15 SAKAI_USER_ID_MAP
>>>     15 SAKAI_USER
>>>     13 SAKAI_PRESENCE
>>>     12 melete_resource
>>>     12 jforum_posts
>>>     11 SAKAI_REALM_PROPERTY
>>>     10 SAKAI_PREFERENCES
>>>     10 CM_MEMBER_CONTAINER_T
>>>      8 CALENDAR_CALENDAR
>>>      7 SAM_ASSESSMENTGRADING_T
>>>      7 CALENDAR_EVENT
>>>      6 the
>>>      6 identifications
>>>      5 melete_section
>>>      5 SST_SITEVISITS
>>>      5 SST_SITEACTIVITY
>>>      4 rutgers_user_section
>>>      4 osp_site_tool
>>>      4 osp_presentation_template
>>>      4 osp_authz_simple
>>>      4 magazines
>>>      4 jforum_import
>>>      3 steps
>>>      3 jforum_topics_mark
>>>      3 content_resource_lock
>>>      3 CHAT2_CHANNEL
>>>      2 osp_template_file_ref
>>>      2 osp_presentation_item_def
>>>      2 metaobj_form_def
>>>      2 melete_module
>>>      2 melete_course_module
>>>      2 jforum_users
>>>      2 jforum_sakai_sessions
>>>      2 a
>>>      2 SST_RESOURCES
>>>      2 SAM_PUBLISHEDEVALUATION_T
>>>      2 SAKAI_SITE_USER
>>>      2 SAKAI_REALM_PROVIDER
>>>      2 SAKAI_REALM
>>>      2 SAKAI_EVENT_DELAY
>>>      2 CONTENT_TYPE_REGISTRY
>>>      2 CM_ACADEMIC_SESSION_T
>>>      2 CHAT2_MESSAGE
>>>      2 ASSIGNMENT_CONTENT
>>>      2 ASSIGNMENT_ASSIGNMENT
>>>      1 osp_presentation_item
>>>      1 osp_presentation_comment
>>>      1 osp_presentation
>>>      1 melete_site_preference
>>>      1 melete_section_resource
>>>      1 melete_module_shdates
>>>      1 jforum_topics
>>>      1 jforum_sessions
>>>      1 jforum_sakai_course_initvalues
>>>      1 jforum_sakai_course_categories
>>>      1 jforum_privmsgs
>>>      1 jforum_categories
>>>      1 SITEASSOC_CONTEXT_ASSOCIATION
>>>      1 SAM_STUDENTGRADINGSUMMARY_T
>>>      1 SAM_PUBLISHEDSECTION_T
>>>      1 SAM_PUBLISHEDFEEDBACK_T
>>>      1 SAM_PUBLISHEDASSESSMENT_T
>>>      1 SAKAI_SYLLABUS_DATA
>>>      1 SAKAI_REALM,
>>>      1 QRTZ_TRIGGERS
>>>      1 CM_SEC_CATEGORY_T
>>> 
>>> 
>>> _______________________________________________
>>> production mailing list
>>> production at collab.sakaiproject.org
>>> http://collab.sakaiproject.org/mailman/listinfo/production
>>> 
>>> TO UNSUBSCRIBE: send email to production-unsubscribe at collab.sakaiproject.org with a subject of "unsubscribe"
>>> 
>>> 
>>> _______________________________________________
>>> production mailing list
>>> production at collab.sakaiproject.org
>>> http://collab.sakaiproject.org/mailman/listinfo/production
>>> 
>>> TO UNSUBSCRIBE: send email to production-unsubscribe at collab.sakaiproject.org with a subject of "unsubscribe"
>> _______________________________________________
>> production mailing list
>> production at collab.sakaiproject.org
>> http://collab.sakaiproject.org/mailman/listinfo/production
>> 
>> TO UNSUBSCRIBE: send email to production-unsubscribe at collab.sakaiproject.org with a subject of "unsubscribe"
> 
> _______________________________________________
> production mailing list
> production at collab.sakaiproject.org
> http://collab.sakaiproject.org/mailman/listinfo/production
> 
> TO UNSUBSCRIBE: send email to production-unsubscribe at collab.sakaiproject.org with a subject of "unsubscribe"

-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://collab.sakaiproject.org/pipermail/production/attachments/20100903/53a0aa71/attachment.html 