[Building Sakai] Gradebook related production incident

[Zhen Qian]

I think the solution is to cache the result of getCategoriesWithStats()
call. See the following call of this method, it seems like none of
parameters is related to specific student...

        List categoriesPlusCG = getCategoriesWithStats(gradebookId,
assignmentSort,

                assignAscending, categorySort, categoryAscending,
gradeRecords, allAssignments);


Thanks,


- Zhen


On Tue, Apr 29, 2014 at 9:45 AM, Steve Swinsburg
<steve.swinsburg at gmail.com>wrote:

> Actually it does look like it does that, and would be congruent with
> Zhen’s numbers (100K items total, 1800 students, 70 grade book items). It
> was probably implemented that way to reduce the number of db calls, but its
> not scalable.
>
> Anyway, a cursory look through the code shows that the culprit may be the
> frontend beans calling some of the API calls in gradebookManager,
> specifically those that take a list of studentUids but end up being called
> with ALL studentUids, like getAllAssignmentGradeRecords et al. I’m not
> familiar enough with that bit of code to be certain thats the issue, but a
> starting point.
>
> Hope that helps.
>
> cheers,
> Steve
>
>
>
>
> On 29 Apr 2014, at 11:09 pm, Steve Swinsburg <steve.swinsburg at gmail.com>
> wrote:
>
> I'm working on a grade book related piece of code right now (full site
> export, which should theoretically have the same performance issue except
> not dealing with logins), but from what Zhen said:
>
> "Upon every student login, the base GB query gets all students grade
> records for all GB items in site (count=100K)."
>
> Does that mean that if both John and Mary are in a site, and John logs
> in, the query gets records for Mary as well? Why? This problem is
> inevitable as the number of students increases. Surely that should be at
> least per context (i.e. lookup for each student) and then cached. Sounds
> like a fairly major change but would be something to start looking at.
>
> If my interpretation is wrong, and it's already per context, could a cache
> be added in between? Can the queries be tuned/return less data and lazy
> load the rest?
>
> cheers,
> Steve
>
>
> On Tue, Apr 29, 2014 at 11:00 PM, Kirschner, Beth <bkirschn at umich.edu>wrote:
>
>> If anyone who's recently worked in the Gradebook code has any suggestions
>> on classes/methods to look at for this problem, please contact me or Zhen
>> off-list. This seems like a risk for anyone that uses Gradebook in large
>> classes.
>>
>> Thanks,
>> - Beth
>>
>> Begin forwarded message:
>>
>> *From: *Zhen Qian <zqian at umich.edu>
>> *Subject: **[Building Sakai] Gradebook related production incident*
>> *Date: *April 25, 2014 4:24:24 PM EDT
>> *To: *sakai-dev <sakai-dev at collab.sakaiproject.org>
>>
>> Hi, all:
>>
>> We had a production issue with Gradebook usage recently, and filed
>> SAK-26153 <https://jira.sakaiproject.org/browse/SAK-26153> for it.
>>
>> In summary, we had a large course site (1800 users) with many categorized
>> Gradebook entries (70 GB items). Instructor sent out "grade available"
>> announcement, and students all log in around the same time to check the
>> final course grade.
>>
>> Upon every student login, the base GB query gets all students grade
>> records for all GB items in site (count=100K). The problem is that there is
>> NO caching implemented, so this query quickly ate up all available db
>> connection pool and brought down app servers.
>>
>> We are running a customized branch based on Gradebook 2.9.2.
>>
>> Has anyone seem similar problems in production? I looked at the latest
>> trunk code for Gradebook, and did not find any recent caching fixes, either.
>>
>> Thanks,
>>
>> - Zhen
>> _______________________________________________
>> sakai-dev mailing list
>> sakai-dev at collab.sakaiproject.org
>> http://collab.sakaiproject.org/mailman/listinfo/sakai-dev
>>
>> TO UNSUBSCRIBE: send email to
>> sakai-dev-unsubscribe at collab.sakaiproject.org with a subject of
>> "unsubscribe"
>>
>>
>>
>> _______________________________________________
>> sakai-dev mailing list
>> sakai-dev at collab.sakaiproject.org
>> http://collab.sakaiproject.org/mailman/listinfo/sakai-dev
>>
>> TO UNSUBSCRIBE: send email to
>> sakai-dev-unsubscribe at collab.sakaiproject.org with a subject of
>> "unsubscribe"
>>
>
>
>
> _______________________________________________
> sakai-dev mailing list
> sakai-dev at collab.sakaiproject.org
> http://collab.sakaiproject.org/mailman/listinfo/sakai-dev
>
> TO UNSUBSCRIBE: send email to
> sakai-dev-unsubscribe at collab.sakaiproject.org with a subject of
> "unsubscribe"
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://collab.sakaiproject.org/pipermail/sakai-dev/attachments/20140429/ecc6cbc2/attachment.html