[Building Sakai] Gradebook related production incident

[Steve Swinsburg]

Actually it does look like it does that, and would be congruent with Zhen’s numbers (100K items total, 1800 students, 70 grade book items). It was probably implemented that way to reduce the number of db calls, but its not scalable.

Anyway, a cursory look through the code shows that the culprit may be the frontend beans calling some of the API calls in gradebookManager, specifically those that take a list of studentUids but end up being called with ALL studentUids, like getAllAssignmentGradeRecords et al. I’m not familiar enough with that bit of code to be certain thats the issue, but a starting point.

Hope that helps.

cheers,
Steve



On 29 Apr 2014, at 11:09 pm, Steve Swinsburg <steve.swinsburg at gmail.com> wrote:

> I'm working on a grade book related piece of code right now (full site export, which should theoretically have the same performance issue except not dealing with logins), but from what Zhen said:
> 
> "Upon every student login, the base GB query gets all students grade records for all GB items in site (count=100K)."
> 
> Does that mean that if both John and Mary are in a site, and John logs in, the query gets records for Mary as well? Why? This problem is inevitable as the number of students increases. Surely that should be at least per context (i.e. lookup for each student) and then cached. Sounds like a fairly major change but would be something to start looking at.
> 
> If my interpretation is wrong, and it's already per context, could a cache be added in between? Can the queries be tuned/return less data and lazy load the rest?
> 
> cheers,
> Steve
> 
> 
> On Tue, Apr 29, 2014 at 11:00 PM, Kirschner, Beth <bkirschn at umich.edu> wrote:
> If anyone who's recently worked in the Gradebook code has any suggestions on classes/methods to look at for this problem, please contact me or Zhen off-list. This seems like a risk for anyone that uses Gradebook in large classes. 
> 
> Thanks,
> - Beth
> 
> Begin forwarded message:
> 
>> From: Zhen Qian <zqian at umich.edu>
>> Subject: [Building Sakai] Gradebook related production incident
>> Date: April 25, 2014 4:24:24 PM EDT
>> To: sakai-dev <sakai-dev at collab.sakaiproject.org>
>> 
>> Hi, all:
>> 
>> We had a production issue with Gradebook usage recently, and filed SAK-26153 for it.
>> 
>> In summary, we had a large course site (1800 users) with many categorized Gradebook entries (70 GB items). Instructor sent out "grade available" announcement, and students all log in around the same time to check the final course grade. 
>> 
>> Upon every student login, the base GB query gets all students grade records for all GB items in site (count=100K). The problem is that there is NO caching implemented, so this query quickly ate up all available db connection pool and brought down app servers.
>> 
>> We are running a customized branch based on Gradebook 2.9.2. 
>> 
>> Has anyone seem similar problems in production? I looked at the latest trunk code for Gradebook, and did not find any recent caching fixes, either.
>> 
>> Thanks,
>> 
>> - Zhen
>> _______________________________________________
>> sakai-dev mailing list
>> sakai-dev at collab.sakaiproject.org
>> http://collab.sakaiproject.org/mailman/listinfo/sakai-dev
>> 
>> TO UNSUBSCRIBE: send email to sakai-dev-unsubscribe at collab.sakaiproject.org with a subject of "unsubscribe"
> 
> 
> _______________________________________________
> sakai-dev mailing list
> sakai-dev at collab.sakaiproject.org
> http://collab.sakaiproject.org/mailman/listinfo/sakai-dev
> 
> TO UNSUBSCRIBE: send email to sakai-dev-unsubscribe at collab.sakaiproject.org with a subject of "unsubscribe"
> 

-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://collab.sakaiproject.org/pipermail/sakai-dev/attachments/20140429/6519ae0a/attachment.html