[Deploying Sakai] 2.7 load

csev csev at umich.edu
Fri Sep 3 05:52:30 PDT 2010


But lets be really clear here - lots of folks are running 2.6 in production with decent performance with KNL-259 in place and no further tweaks.  KNL-259 was identified and fixed very early in the 2.6 process.

The kernel team did not put the patch into trunk - so for 2.7 we had a performance regression.  The fix is to use the code in KNL-259 and hopefully put it into 2-6-x, 2-7-x and trunk if that has not already been done.

What you are proposing in KNL-577 is new work and further performance improvement above and beyond KNL-259 which probably would be a good idea.  Lets just make sure that this work you are doing is not considered part of the "lets fix the 2.7 problem" and part of overall tuning.

That suggests careful testing of your new work before pushing it out since there is no "emergency" or "rush" associated with KNL-577.  Getting KNL-259 into 2.7 (and 2.6 if not already done) production is something we need to get done quickly as it is well tested with lots of production experience for well over a year.

/Chuck

On Sep 3, 2010, at 3:44 AM, David Horwitz wrote:

> Looking at the code what seems to have happened is that Chuck added caching to BaseDbFlat storage that caches all properties collections accesses that way. The properties in question are accessed directly from the Db so are not cached. I'll have a look at adding some caching here - it should be pretty straight foward. http://jira.sakaiproject.org/browse/KNL-577
> 
> D
> 
> On 09/02/2010 08:47 PM, Matthew Jones wrote:
>> 
>> Well, the SITE_PAGE_PROPERTY should be fixed by the issue KNL-259 that Noah/Zhen worked on. It didn't get committed to the main sakai repo for inclusing until 2.7.1/2.6.3, I'm betting this will help out a little. This was run both at IU and Michigan and resulted both in decreased queries and increased caching of these properties.
>> 
>> Not sure about that TOOL_PROPERTY one but it "seems" like that one should/could be cached as well. I don't see one specifically set up for it though in the bean in db-components.xml or displaying in the memory tool.
>> 
>> (Sample cache report)
>> org.sakaiproject.db.BaseDbFlatStorage.SAKAI_ALIAS_PROPERTY: count:0 hits:34 misses:34 hit%:50
>> org.sakaiproject.db.BaseDbFlatStorage.SAKAI_REALM_PROPERTY: count:85 hits:180440 misses:59292 hit%:75
>> org.sakaiproject.db.BaseDbFlatStorage.SAKAI_SITE_GROUP_PROPERTY: count:0 hits:0 misses:24474 hit%:0
>> org.sakaiproject.db.BaseDbFlatStorage.SAKAI_SITE_PAGE_PROPERTY: count:374 hits:12941 misses:17298 hit%:42
>> org.sakaiproject.db.BaseDbFlatStorage.SAKAI_SITE_PROPERTY: count:329 hits:725659 misses:187371 hit%:79
>> org.sakaiproject.db.BaseDbFlatStorage.SAKAI_USER_PROPERTY: count:0 hits:12 misses:12 hit%:50
>> 
>> Stephen mentioned back in this JIRA [1] he also didn't notice this SAKAI_SITE_TOOL_PROPERTY show up in the cache list either. Would be worth investigating if this is possible, why it wasn't done.
>> 
>> In talking to Chuck S, he recommended that kernel property (and other) caching be completely reviewed and improved, but this sounds like a big (2.9, 2.10) job.
>> 
>> In Glenn's only blog post (that I know of) [2], he talked about improving the caching and performance of all of those SAKAI_REALM functions. Those hit us pretty hard too (with realm queries and presence driving most of the database traffic). Especially on some sites with lots of users and resources in content.
>> 
>> [1] http://jira.sakaiproject.org/browse/SAK-13886?focusedCommentId=65873&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#action_65873
>> 
>> [2] http://galvanometer.blogspot.com/
>> 
>> On Thu, Sep 2, 2010 at 8:47 AM, Hedrick Charles <hedrick at rutgers.edu> wrote:
>> The students just got back for the Fall, so this is the first time we've run 2.7 under a realistic load. It looks to me like the CPU usage on both front ends and database is significantly higher, by at least a factor of 2. This is probably not going to kill us, but it worries me.
>> 
>> I took a very brief snapshot of database queries during the day yesterday. Here's the tables it was querying. The scripts were pretty simple-minded, so this isn't perfect. But it should give an indication. The top queries seem to be simple lookups on single tables, so caching should work very well. I see no evidence that this data is actually being cached.
>> 
>> 
>>   2561 SAKAI_SITE_TOOL_PROPERTY
>>   2196 SAKAI_SITE_TOOL
>>   1237 SAKAI_REALM_RL_FN,SAKAI_REALM
>>   1053 SAKAI_SITE_PAGE_PROPERTY
>>    253 SAKAI_SITE_PAGE
>>    237
>>    181 SAKAI_REALM_ROLE
>>    108 SAKAI_PRIVACY_RECORD
>>     89 SAKAI_SITE,SAKAI_SITE_USER
>>     84 CONTENT_COLLECTION
>>     70 SAKAI_SITE_PROPERTY
>>     64 SAKAI_SITE
>>     63 CONTENT_RESOURCE
>>     48 SAKAI_REALM_ROLE_DESC
>>     48 SAKAI_REALM_RL_GR
>>     48 SAKAI_REALM_RL_FN
>>     47 rutgers_user
>>     33 SAKAI_ALIAS
>>     30 SAKAI_SYLLABUS_ITEM
>>     24 SAKAI_SESSION
>>     24 SAKAI_REALM_FUNCTION
>>     22 SAKAI_SITE_GROUP_PROPERTY
>>     22 SAKAI_SITE_GROUP
>>     20 ANNOUNCEMENT_CHANNEL
>>     19 ANNOUNCEMENT_MESSAGE
>>     15 SST_EVENTS
>>     15 SAKAI_USER_ID_MAP
>>     15 SAKAI_USER
>>     13 SAKAI_PRESENCE
>>     12 melete_resource
>>     12 jforum_posts
>>     11 SAKAI_REALM_PROPERTY
>>     10 SAKAI_PREFERENCES
>>     10 CM_MEMBER_CONTAINER_T
>>      8 CALENDAR_CALENDAR
>>      7 SAM_ASSESSMENTGRADING_T
>>      7 CALENDAR_EVENT
>>      6 the
>>      6 identifications
>>      5 melete_section
>>      5 SST_SITEVISITS
>>      5 SST_SITEACTIVITY
>>      4 rutgers_user_section
>>      4 osp_site_tool
>>      4 osp_presentation_template
>>      4 osp_authz_simple
>>      4 magazines
>>      4 jforum_import
>>      3 steps
>>      3 jforum_topics_mark
>>      3 content_resource_lock
>>      3 CHAT2_CHANNEL
>>      2 osp_template_file_ref
>>      2 osp_presentation_item_def
>>      2 metaobj_form_def
>>      2 melete_module
>>      2 melete_course_module
>>      2 jforum_users
>>      2 jforum_sakai_sessions
>>      2 a
>>      2 SST_RESOURCES
>>      2 SAM_PUBLISHEDEVALUATION_T
>>      2 SAKAI_SITE_USER
>>      2 SAKAI_REALM_PROVIDER
>>      2 SAKAI_REALM
>>      2 SAKAI_EVENT_DELAY
>>      2 CONTENT_TYPE_REGISTRY
>>      2 CM_ACADEMIC_SESSION_T
>>      2 CHAT2_MESSAGE
>>      2 ASSIGNMENT_CONTENT
>>      2 ASSIGNMENT_ASSIGNMENT
>>      1 osp_presentation_item
>>      1 osp_presentation_comment
>>      1 osp_presentation
>>      1 melete_site_preference
>>      1 melete_section_resource
>>      1 melete_module_shdates
>>      1 jforum_topics
>>      1 jforum_sessions
>>      1 jforum_sakai_course_initvalues
>>      1 jforum_sakai_course_categories
>>      1 jforum_privmsgs
>>      1 jforum_categories
>>      1 SITEASSOC_CONTEXT_ASSOCIATION
>>      1 SAM_STUDENTGRADINGSUMMARY_T
>>      1 SAM_PUBLISHEDSECTION_T
>>      1 SAM_PUBLISHEDFEEDBACK_T
>>      1 SAM_PUBLISHEDASSESSMENT_T
>>      1 SAKAI_SYLLABUS_DATA
>>      1 SAKAI_REALM,
>>      1 QRTZ_TRIGGERS
>>      1 CM_SEC_CATEGORY_T
>> 
>> 
>> _______________________________________________
>> production mailing list
>> production at collab.sakaiproject.org
>> http://collab.sakaiproject.org/mailman/listinfo/production
>> 
>> TO UNSUBSCRIBE: send email to production-unsubscribe at collab.sakaiproject.org with a subject of "unsubscribe"
>> 
>> 
>> _______________________________________________
>> production mailing list
>> production at collab.sakaiproject.org
>> http://collab.sakaiproject.org/mailman/listinfo/production
>> 
>> TO UNSUBSCRIBE: send email to production-unsubscribe at collab.sakaiproject.org with a subject of "unsubscribe"
> _______________________________________________
> production mailing list
> production at collab.sakaiproject.org
> http://collab.sakaiproject.org/mailman/listinfo/production
> 
> TO UNSUBSCRIBE: send email to production-unsubscribe at collab.sakaiproject.org with a subject of "unsubscribe"

-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://collab.sakaiproject.org/pipermail/production/attachments/20100903/703fecd8/attachment.html 


More information about the production mailing list