[Building Sakai] Fwd: Gradebook related production incident

[Steve Swinsburg]

I'm working on a grade book related piece of code right now (full site
export, which should theoretically have the same performance issue except
not dealing with logins), but from what Zhen said:

"Upon every student login, the base GB query gets all students grade
records for all GB items in site (count=100K)."

Does that mean that if both John and Mary are in a site, and John logs in,
the query gets records for Mary as well? Why? This problem is inevitable as
the number of students increases. Surely that should be at least per
context (i.e. lookup for each student) and then cached. Sounds like a
fairly major change but would be something to start looking at.

If my interpretation is wrong, and it's already per context, could a cache
be added in between? Can the queries be tuned/return less data and lazy
load the rest?

cheers,
Steve


On Tue, Apr 29, 2014 at 11:00 PM, Kirschner, Beth <bkirschn at umich.edu>wrote:

> If anyone who's recently worked in the Gradebook code has any suggestions
> on classes/methods to look at for this problem, please contact me or Zhen
> off-list. This seems like a risk for anyone that uses Gradebook in large
> classes.
>
> Thanks,
> - Beth
>
> Begin forwarded message:
>
> *From: *Zhen Qian <zqian at umich.edu>
> *Subject: **[Building Sakai] Gradebook related production incident*
> *Date: *April 25, 2014 4:24:24 PM EDT
> *To: *sakai-dev <sakai-dev at collab.sakaiproject.org>
>
> Hi, all:
>
> We had a production issue with Gradebook usage recently, and filed
> SAK-26153 <https://jira.sakaiproject.org/browse/SAK-26153> for it.
>
> In summary, we had a large course site (1800 users) with many categorized
> Gradebook entries (70 GB items). Instructor sent out "grade available"
> announcement, and students all log in around the same time to check the
> final course grade.
>
> Upon every student login, the base GB query gets all students grade
> records for all GB items in site (count=100K). The problem is that there is
> NO caching implemented, so this query quickly ate up all available db
> connection pool and brought down app servers.
>
> We are running a customized branch based on Gradebook 2.9.2.
>
> Has anyone seem similar problems in production? I looked at the latest
> trunk code for Gradebook, and did not find any recent caching fixes, either.
>
> Thanks,
>
> - Zhen
> _______________________________________________
> sakai-dev mailing list
> sakai-dev at collab.sakaiproject.org
> http://collab.sakaiproject.org/mailman/listinfo/sakai-dev
>
> TO UNSUBSCRIBE: send email to
> sakai-dev-unsubscribe at collab.sakaiproject.org with a subject of
> "unsubscribe"
>
>
>
> _______________________________________________
> sakai-dev mailing list
> sakai-dev at collab.sakaiproject.org
> http://collab.sakaiproject.org/mailman/listinfo/sakai-dev
>
> TO UNSUBSCRIBE: send email to
> sakai-dev-unsubscribe at collab.sakaiproject.org with a subject of
> "unsubscribe"
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://collab.sakaiproject.org/pipermail/sakai-dev/attachments/20140429/a062a91e/attachment.html