[Deploying Sakai] 2.7 load
David Horwitz
david.horwitz at uct.ac.za
Fri Sep 3 01:54:40 PDT 2010
Ok Looking at this I have a couple of observations (note these are based
on code review not profiling or anything).
1) Tool Properties will suffer from the same problem as page properties
did prior to KNL-259
2) The calls for all site properties (e.g All the SITE_PAGE_PROPERTIES
in !admin) will bypass the caches as the caching is done by DbFlatStorage)
3) it suggested that DbStorage.readToolProperties is never called
It looks to me like fixing this will entail some moderate code
refactor, probably to move the caching to DbSiteService
D
On 09/03/2010 09:44 AM, David Horwitz wrote:
> Looking at the code what seems to have happened is that Chuck added
> caching to BaseDbFlat storage that caches all properties collections
> accesses that way. The properties in question are accessed directly
> from the Db so are not cached. I'll have a look at adding some caching
> here - it should be pretty straight foward.
> http://jira.sakaiproject.org/browse/KNL-577
>
> D
>
> On 09/02/2010 08:47 PM, Matthew Jones wrote:
>> Well, the SITE_PAGE_PROPERTY should be fixed by the issue KNL-259
>> that Noah/Zhen worked on. It didn't get committed to the main sakai
>> repo for inclusing until 2.7.1/2.6.3, I'm betting this will help out
>> a little. This was run both at IU and Michigan and resulted both in
>> decreased queries and increased caching of these properties.
>>
>> Not sure about that TOOL_PROPERTY one but it "seems" like that one
>> should/could be cached as well. I don't see one specifically set up
>> for it though in the bean in db-components.xml or displaying in the
>> memory tool.
>>
>> (Sample cache report)
>> org.sakaiproject.db.BaseDbFlatStorage.SAKAI_ALIAS_PROPERTY: count:0
>> hits:34 misses:34 hit%:50
>> org.sakaiproject.db.BaseDbFlatStorage.SAKAI_REALM_PROPERTY: count:85
>> hits:180440 misses:59292 hit%:75
>> org.sakaiproject.db.BaseDbFlatStorage.SAKAI_SITE_GROUP_PROPERTY:
>> count:0 hits:0 misses:24474 hit%:0
>> org.sakaiproject.db.BaseDbFlatStorage.SAKAI_SITE_PAGE_PROPERTY:
>> count:374 hits:12941 misses:17298 hit%:42
>> org.sakaiproject.db.BaseDbFlatStorage.SAKAI_SITE_PROPERTY: count:329
>> hits:725659 misses:187371 hit%:79
>> org.sakaiproject.db.BaseDbFlatStorage.SAKAI_USER_PROPERTY: count:0
>> hits:12 misses:12 hit%:50
>>
>> Stephen mentioned back in this JIRA [1] he also didn't notice this
>> SAKAI_SITE_TOOL_PROPERTY show up in the cache list either. Would be
>> worth investigating if this is possible, why it wasn't done.
>>
>> In talking to Chuck S, he recommended that kernel property (and
>> other) caching be completely reviewed and improved, but this sounds
>> like a big (2.9, 2.10) job.
>>
>> In Glenn's only blog post (that I know of) [2], he talked about
>> improving the caching and performance of all of those SAKAI_REALM
>> functions. Those hit us pretty hard too (with realm queries and
>> presence driving most of the database traffic). Especially on some
>> sites with lots of users and resources in content.
>>
>> [1] http://jira.sakaiproject.org/browse/SAK-13886?focusedCommentId=65873&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#action_65873
>> <http://jira.sakaiproject.org/browse/SAK-13886?focusedCommentId=65873&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#action_65873>
>>
>> [2] http://galvanometer.blogspot.com/
>>
>> On Thu, Sep 2, 2010 at 8:47 AM, Hedrick Charles <hedrick at rutgers.edu
>> <mailto:hedrick at rutgers.edu>> wrote:
>>
>> The students just got back for the Fall, so this is the first
>> time we've run 2.7 under a realistic load. It looks to me like
>> the CPU usage on both front ends and database is significantly
>> higher, by at least a factor of 2. This is probably not going to
>> kill us, but it worries me.
>>
>> I took a very brief snapshot of database queries during the day
>> yesterday. Here's the tables it was querying. The scripts were
>> pretty simple-minded, so this isn't perfect. But it should give
>> an indication. The top queries seem to be simple lookups on
>> single tables, so caching should work very well. I see no
>> evidence that this data is actually being cached.
>>
>>
>> 2561 SAKAI_SITE_TOOL_PROPERTY
>> 2196 SAKAI_SITE_TOOL
>> 1237 SAKAI_REALM_RL_FN,SAKAI_REALM
>> 1053 SAKAI_SITE_PAGE_PROPERTY
>> 253 SAKAI_SITE_PAGE
>> 237
>> 181 SAKAI_REALM_ROLE
>> 108 SAKAI_PRIVACY_RECORD
>> 89 SAKAI_SITE,SAKAI_SITE_USER
>> 84 CONTENT_COLLECTION
>> 70 SAKAI_SITE_PROPERTY
>> 64 SAKAI_SITE
>> 63 CONTENT_RESOURCE
>> 48 SAKAI_REALM_ROLE_DESC
>> 48 SAKAI_REALM_RL_GR
>> 48 SAKAI_REALM_RL_FN
>> 47 rutgers_user
>> 33 SAKAI_ALIAS
>> 30 SAKAI_SYLLABUS_ITEM
>> 24 SAKAI_SESSION
>> 24 SAKAI_REALM_FUNCTION
>> 22 SAKAI_SITE_GROUP_PROPERTY
>> 22 SAKAI_SITE_GROUP
>> 20 ANNOUNCEMENT_CHANNEL
>> 19 ANNOUNCEMENT_MESSAGE
>> 15 SST_EVENTS
>> 15 SAKAI_USER_ID_MAP
>> 15 SAKAI_USER
>> 13 SAKAI_PRESENCE
>> 12 melete_resource
>> 12 jforum_posts
>> 11 SAKAI_REALM_PROPERTY
>> 10 SAKAI_PREFERENCES
>> 10 CM_MEMBER_CONTAINER_T
>> 8 CALENDAR_CALENDAR
>> 7 SAM_ASSESSMENTGRADING_T
>> 7 CALENDAR_EVENT
>> 6 the
>> 6 identifications
>> 5 melete_section
>> 5 SST_SITEVISITS
>> 5 SST_SITEACTIVITY
>> 4 rutgers_user_section
>> 4 osp_site_tool
>> 4 osp_presentation_template
>> 4 osp_authz_simple
>> 4 magazines
>> 4 jforum_import
>> 3 steps
>> 3 jforum_topics_mark
>> 3 content_resource_lock
>> 3 CHAT2_CHANNEL
>> 2 osp_template_file_ref
>> 2 osp_presentation_item_def
>> 2 metaobj_form_def
>> 2 melete_module
>> 2 melete_course_module
>> 2 jforum_users
>> 2 jforum_sakai_sessions
>> 2 a
>> 2 SST_RESOURCES
>> 2 SAM_PUBLISHEDEVALUATION_T
>> 2 SAKAI_SITE_USER
>> 2 SAKAI_REALM_PROVIDER
>> 2 SAKAI_REALM
>> 2 SAKAI_EVENT_DELAY
>> 2 CONTENT_TYPE_REGISTRY
>> 2 CM_ACADEMIC_SESSION_T
>> 2 CHAT2_MESSAGE
>> 2 ASSIGNMENT_CONTENT
>> 2 ASSIGNMENT_ASSIGNMENT
>> 1 osp_presentation_item
>> 1 osp_presentation_comment
>> 1 osp_presentation
>> 1 melete_site_preference
>> 1 melete_section_resource
>> 1 melete_module_shdates
>> 1 jforum_topics
>> 1 jforum_sessions
>> 1 jforum_sakai_course_initvalues
>> 1 jforum_sakai_course_categories
>> 1 jforum_privmsgs
>> 1 jforum_categories
>> 1 SITEASSOC_CONTEXT_ASSOCIATION
>> 1 SAM_STUDENTGRADINGSUMMARY_T
>> 1 SAM_PUBLISHEDSECTION_T
>> 1 SAM_PUBLISHEDFEEDBACK_T
>> 1 SAM_PUBLISHEDASSESSMENT_T
>> 1 SAKAI_SYLLABUS_DATA
>> 1 SAKAI_REALM,
>> 1 QRTZ_TRIGGERS
>> 1 CM_SEC_CATEGORY_T
>>
>>
>> _______________________________________________
>> production mailing list
>> production at collab.sakaiproject.org
>> <mailto:production at collab.sakaiproject.org>
>> http://collab.sakaiproject.org/mailman/listinfo/production
>>
>> TO UNSUBSCRIBE: send email to
>> production-unsubscribe at collab.sakaiproject.org
>> <mailto:production-unsubscribe at collab.sakaiproject.org> with a
>> subject of "unsubscribe"
>>
>>
>>
>> _______________________________________________
>> production mailing list
>> production at collab.sakaiproject.org
>> http://collab.sakaiproject.org/mailman/listinfo/production
>>
>> TO UNSUBSCRIBE: send email to production-unsubscribe at collab.sakaiproject.org with a subject of "unsubscribe"
>
>
> _______________________________________________
> production mailing list
> production at collab.sakaiproject.org
> http://collab.sakaiproject.org/mailman/listinfo/production
>
> TO UNSUBSCRIBE: send email to production-unsubscribe at collab.sakaiproject.org with a subject of "unsubscribe"
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://collab.sakaiproject.org/pipermail/production/attachments/20100903/02ff20a2/attachment-0001.html
More information about the production
mailing list