[samigo-team] SAM-2102 small hibernate patch

[David Horwitz]

Great work Sam!

I will have a look at the 2 Jiras (yesterday was a holiday here) But my first glance of both looks good.

A thought is that these are both issues that could affect other hibernate tools, it might be useful to float to a wider group.

D

On Thu, 2013-03-21 at 13:50 -0400, Sam Ottenhoff wrote:
Sorry here is the new JIRA: https://jira.sakaiproject.org/browse/SAM-2107


On Thu, Mar 21, 2013 at 1:50 PM, Sam Ottenhoff <ottenhoff at longsight.com<mailto:ottenhoff at longsight.com>> wrote:
Okay, it seems almost too good to be true so I'm hoping others can review.... I now see a 20x decrease in queries after a patch I just attached to SAM-2107.  I am still using the same assessment with 8 parts, 2 questions per part from a pool of 42.  Samigo trunk requires 9886 queries.  Samigo with SAM-2102 and SAM-2107 requires 434 queries.


The problems with SAM_PUBLISHEDITEMMETADATA_T and SAM_PUBLISHEDITEMFEEDBACK_T seemed to be related to helper methods that converted the Java Sets to HashMaps for convenience.  Unfortunately, the conversion to HashMaps turned into thousands of unnecessary Hibernate retrievals.  So I just removed the HashMap helper methods and replaced with simple iterations over the Sets (the Set is already in memory so no more database queries).


--Sam



On Thu, Mar 21, 2013 at 11:57 AM, Sam Ottenhoff <ottenhoff at longsight.com<mailto:ottenhoff at longsight.com>> wrote:
The batch size doesn't seem to matter much as long as it is over 10 or so.  The basic problem is that Samigo uses lots of one-to-many relationships to go from one assessment => many sections => many items  (+ feedback/metadata) => many answers (+feedback).  So the batch seems to allow getting the "many" items all at once instead of individually.   So as long as the batch-size is large enough to get all answers for an item or all items for a section, it should be fine.  50 seemed safe and reasonable to me.


I still can't seem to reduce the queries to SAM_PUBLISHEDITEMMETADATA_T and SAM_PUBLISHEDITEMFEEDBACK_T.  Samigo seems to load item for every single possible question at both assessment begin and assessment end.  Does anyone know how/why they are consulted at the end of the assessment?  That might be a quick way to cut query volume by another 50% or so.


--Sam



On Thu, Mar 21, 2013 at 12:58 AM, Karen Tsao <ktsao at stanford.edu<mailto:ktsao at stanford.edu>> wrote:
The patch looks good to me.


Just have a quick question. How do you decide the batch size?


Thanks,
Karen

On Wed, Mar 20, 2013 at 1:28 PM, Sam Ottenhoff <ottenhoff at longsight.com<mailto:ottenhoff at longsight.com>> wrote:

On the latest trunk, taking a 16-question random draw (2 questions each in 8 parts from a 42-question pool) assessment generates about 9886 queries to the database.


I uploaded a small patch to the hibernate mapping file that reduced the number of queries for my one attempt down to 1539 queries.


Can someone review?


https://jira.sakaiproject.org/browse/SAM-2102


--Sam


_______________________________________________
samigo-team mailing list
samigo-team at collab.sakaiproject.org<mailto:samigo-team at collab.sakaiproject.org>
http://collab.sakaiproject.org/mailman/listinfo/samigo-team











_______________________________________________
samigo-team mailing list
samigo-team at collab.sakaiproject.org<mailto:samigo-team at collab.sakaiproject.org>
http://collab.sakaiproject.org/mailman/listinfo/samigo-team


________________________________
UNIVERSITY OF CAPE TOWN

This e-mail is subject to the UCT ICT policies and e-mail disclaimer published on our website at http://www.uct.ac.za/about/policies/emaildisclaimer/ or obtainable from +27 21 650 9111. This e-mail is intended only for the person(s) to whom it is addressed. If the e-mail has reached you in error, please notify the author. If you are not the intended recipient of the e-mail you may not use, disclose, copy, redirect or print the content. If this e-mail is not related to the business of UCT it is sent by the sender in the sender's individual capacity.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://collab.sakaiproject.org/pipermail/samigo-team/attachments/20130322/faa84559/attachment.html