[Building Sakai] Production issue with Lesson Builder

John Bush jbush at anisakai.com
Tue Jan 21 23:06:57 PST 2014 

We had similar issue back in October but with the announcement tool.  An
instructor got what I like to call all "clicky with it".  He literally
figured out a way to create 3000+ announcements.  I'm pretty sure it was
some type of anger management issue because the request didn't come back
fast enough for him, so hit just kept clicking away.

We tracked him down gave him some therapy, and it got better.  But we were
in a pinch and worried about releasing the Longsight patch (which sounds
solid) without much time to test it out, so we simply patched in some stuff
into the velocity code to gray out the button after you click it once.

So even if this instructor fell off the anger management wagon again, we'd
have some protection.  I could point you towards that patch which is less
involved, but not as thorough as this patch, if you are interested.


On Mon, Jan 13, 2014 at 11:15 AM, Bryan Holladay <holladay at longsight.com>wrote:

> >When did Longsight introduce the code
> I introduced our solution to alleviate the Lesson Builder import problem
> in the beginning of the Fall semester last year.  The jira was created on
> Sept 4th 2013.  We probably ran into the bug a few days prior to that.  We
> notified our clients and updated the ones who were willing to update, so
> the majority of our 2.9 instances have this patch.  We have several people
> starting their first week this week, but I don't have a number for that.
>
> Once again, this patch doesn't address LB specifically and is more about a
> larger problem with site import running in the user's thread.  However, it
> was the LB import that we saw causing the biggest issue and was the
> catalyst for this, which is why the jira only briefly mentions LB.  I would
> love to hear what Chuck H. has to add about the LB import code itself.
>
> -Bryan
>
>
> On Mon, Jan 13, 2014 at 12:25 PM, May, Megan Marie <mmmay at iu.edu> wrote:
>
>>  Our testing didn’t show that LSNBLDR-304<https://jira.sakaiproject.org/browse/LSNBLDR-304>was the solution.
>>
>>
>>
>> And thanks Bryan.    We also tracked down the faculty member to ask them
>> to stop.  Thank goodness they were a good sport about it!     When did
>> Longsight introduce the code (and have any schools been through the first
>> week or so?
>>
>>
>>
>> Megan
>>
>>
>>
>> *From:* sakai-dev-bounces at collab.sakaiproject.org [mailto:
>> sakai-dev-bounces at collab.sakaiproject.org] *On Behalf Of *Neal Caidin
>> *Sent:* Monday, January 13, 2014 10:51 AM
>> *To:* Kirschner, Beth
>>
>> *Cc:* sakai-dev (sakai-dev at collab.sakaiproject.org)
>> *Subject:* Re: [Building Sakai] Production issue with Lesson Builder
>>
>>
>>
>> Is it related to this one
>>
>> https://jira.sakaiproject.org/browse/LSNBLDR-304   ?
>>
>> -- Neal
>>
>>
>>
>>     *Kirschner, Beth* <bkirschn at umich.edu>
>>
>> January 13, 2014 at 10:45 AM
>>
>> Is this written up in JIRA anywhere?
>>
>> - Beth
>>
>>
>> _______________________________________________
>> sakai-dev mailing list
>> sakai-dev at collab.sakaiproject.org
>> http://collab.sakaiproject.org/mailman/listinfo/sakai-dev
>>
>> TO UNSUBSCRIBE: send email to
>> sakai-dev-unsubscribe at collab.sakaiproject.org with a subject of
>> "unsubscribe"
>>
>>   *Bryan Holladay* <holladay at longsight.com>
>>
>> January 13, 2014 at 10:39 AM
>>
>> >Does that help with the excessive number of items?
>>
>> Yes and no.  It prevents the excessive number of orphaned pages by the
>> instructor re-importing and deleting the half imported material.  I've seen
>> cases where it takes 10+ minutes for the import to complete, which is why
>> instructor's keep trying the import and causing millions of pages to be
>> created.  We actually had to track the instructor down and tell them to
>> stop importing until we get the fix in since every time we restarted the
>> instructor went back in to re-import.  However, this patch doesn't address
>> the issue inside of LB's import code.  So I'm sure there is room for more
>> improvement, but it seemed like it was the combination of multiple imports
>> + non-performant import code.  The good/bad thing about this issue is that
>> it only comes up in the first week or so of the semester.  So it won't be
>> an issue again until next semester and any fix you add you won't really be
>> certain until then as well.  I'm not going to jinx us by saying everything
>> is going fine so far with that patch with this start of this semester but
>> you can infer that if you wish ;)
>>
>>
>>
>> Thanks,
>>
>> Bryan
>>
>>    _______________________________________________
>> sakai-dev mailing list
>> sakai-dev at collab.sakaiproject.org
>> http://collab.sakaiproject.org/mailman/listinfo/sakai-dev
>>
>> TO UNSUBSCRIBE: send email to
>> sakai-dev-unsubscribe at collab.sakaiproject.org with a subject of
>> "unsubscribe"
>>
>>   *May, Megan Marie* <mmmay at iu.edu>
>>
>> January 13, 2014 at 9:16 AM
>>
>> Thanks Bryan!     We started looking at that patch last night.
>>
>>
>>
>> Does that help with the excessive number of items?  Even with 1 import
>> we’ve seen a high number of duplicate entries (over 1000).
>>
>> Megan
>>
>>
>>
>> *From:* Bryan Holladay [mailto:holladay at longsight.com<holladay at longsight.com>]
>>
>> *Sent:* Monday, January 13, 2014 8:38 AM
>> *To:* May, Megan Marie
>> *Cc:* sakai-dev (sakai-dev at collab.sakaiproject.org)
>> *Subject:* Re: [Building Sakai] Production issue with Lesson Builder
>>
>>
>>
>> My first assumption would be that it is from importing previous course
>> LB's into the new courses.  We've seen a lot of issues with performance,
>> excessive number of items and server stress with this process.  We
>> created/provided this jira to help alleviate some of the issue:
>>
>>
>>
>> SAK-23897 Site Info -> Import from site needs to run in a seperate thread
>> and email user when finished
>>
>>
>>
>> We were seeing instructors importing several times b/c it would either
>> time out or they'd give up on the process and try again (or click several
>> times).  If I wanted to bring down a Sakai server, that is how I would do
>> it (site import with LB)
>>
>>
>>
>> -Bryan
>>
>>
>>
>> On Sun, Jan 12, 2014 at 1:22 PM, May, Megan Marie <mmmay at iu.edu> wrote:
>>
>> On Friday the following SQL began running for excessive periods of time
>>
>>
>>
>> update lesson_builder_items set pageId=:1 , sequence=:2 , type=:3 ,
>> sakaiId=:4 , name=:5 , html=:6 , description=:7 , height=:8 , width=:9 ,
>> alt=:10 , nextPage=:11 , format=:12 , required=:13 , alternate=:14 ,
>> prerequisite=:15 , subrequirement=:16 , requirementText=:17 ,
>> sameWindow=:18 , groups=:19 , anonymous=:20 , showComments=:21 ,
>> forcedCommentsAnonymous=:22 , gradebookId=:23 , gradebookPoints=:24 ,
>> gradebookTitle=:25 , altGradebook=:26 , altPoints=:27 ,
>> altGradebookTitle=:28 where id=:29
>>
>>
>>
>> As a stop gap we’ve bounced the app servers these are occurring on.
>>  We’ve noted HUGE increases in the number of rows in LESSON_BUILDER_ITEMS
>> & the increases pertain to  rows in lesson_builder_items show a SAKAIID of
>> “/dummy”  (Import issues?)   The behavior seems like a loop – and we’ve
>> looked at LSNBLDR-304 although the cause of it is ambiguous so we’re
>> hesitant to move forward with it.
>>
>>
>>
>> Has anyone encountered this?
>>
>>
>>
>> Megan
>>
>>
>>
>> Megan May
>>
>> Manager, Learning Management Systems / Kuali Student Development
>>
>> UITS Enterprise Student Systems
>>
>> Enterprise Software Division
>>
>> Indiana University
>>
>> Office:  317-274-4528
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>> _______________________________________________
>> sakai-dev mailing list
>> sakai-dev at collab.sakaiproject.org
>> http://collab.sakaiproject.org/mailman/listinfo/sakai-dev
>>
>> TO UNSUBSCRIBE: send email to
>> sakai-dev-unsubscribe at collab.sakaiproject.org with a subject of
>> "unsubscribe"
>>
>>
>>
>> _______________________________________________
>> sakai-dev mailing list
>> sakai-dev at collab.sakaiproject.org
>> http://collab.sakaiproject.org/mailman/listinfo/sakai-dev
>>
>> TO UNSUBSCRIBE: send email to
>> sakai-dev-unsubscribe at collab.sakaiproject.org with a subject of
>> "unsubscribe"
>>
>>   *Bryan Holladay* <holladay at longsight.com>
>>
>> January 13, 2014 at 8:37 AM
>>
>> My first assumption would be that it is from importing previous course
>> LB's into the new courses.  We've seen a lot of issues with performance,
>> excessive number of items and server stress with this process.  We
>> created/provided this jira to help alleviate some of the issue:
>>
>>
>>
>> SAK-23897 Site Info -> Import from site needs to run in a seperate thread
>> and email user when finished
>>
>>
>>
>> We were seeing instructors importing several times b/c it would either
>> time out or they'd give up on the process and try again (or click several
>> times).  If I wanted to bring down a Sakai server, that is how I would do
>> it (site import with LB)
>>
>>
>>
>> -Bryan
>>
>>
>>
>>   _______________________________________________
>> sakai-dev mailing list
>> sakai-dev at collab.sakaiproject.org
>> http://collab.sakaiproject.org/mailman/listinfo/sakai-dev
>>
>> TO UNSUBSCRIBE: send email to
>> sakai-dev-unsubscribe at collab.sakaiproject.org with a subject of
>> "unsubscribe"
>>
>>   *May, Megan Marie* <mmmay at iu.edu>
>>
>> January 12, 2014 at 1:22 PM
>>
>> On Friday the following SQL began running for excessive periods of time
>>
>>
>>
>> update lesson_builder_items set pageId=:1 , sequence=:2 , type=:3 ,
>> sakaiId=:4 , name=:5 , html=:6 , description=:7 , height=:8 , width=:9 ,
>> alt=:10 , nextPage=:11 , format=:12 , required=:13 , alternate=:14 ,
>> prerequisite=:15 , subrequirement=:16 , requirementText=:17 ,
>> sameWindow=:18 , groups=:19 , anonymous=:20 , showComments=:21 ,
>> forcedCommentsAnonymous=:22 , gradebookId=:23 , gradebookPoints=:24 ,
>> gradebookTitle=:25 , altGradebook=:26 , altPoints=:27 ,
>> altGradebookTitle=:28 where id=:29
>>
>>
>>
>> As a stop gap we’ve bounced the app servers these are occurring on.
>>  We’ve noted HUGE increases in the number of rows in LESSON_BUILDER_ITEMS
>> & the increases pertain to  rows in lesson_builder_items show a SAKAIID
>> of “/dummy”  (Import issues?)   The behavior seems like a loop – and we’ve
>> looked at LSNBLDR-304 although the cause of it is ambiguous so we’re
>> hesitant to move forward with it.
>>
>>
>>
>> Has anyone encountered this?
>>
>>
>>
>> Megan
>>
>>
>>
>> Megan May
>>
>> Manager, Learning Management Systems / Kuali Student Development
>>
>> UITS Enterprise Student Systems
>>
>> Enterprise Software Division
>>
>> Indiana University
>>
>> Office:  317-274-4528
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>> _______________________________________________
>> sakai-dev mailing list
>> sakai-dev at collab.sakaiproject.org
>> http://collab.sakaiproject.org/mailman/listinfo/sakai-dev
>>
>> TO UNSUBSCRIBE: send email to
>> sakai-dev-unsubscribe at collab.sakaiproject.org with a subject of
>> "unsubscribe"
>>
>>
>>
>> --
>> Neal Caidin
>> Sakai Community Coordinator
>> Apereo Foundation
>> neal.caidin at apereo.org
>> Skype me! (but let me know in advance for the first interaction) -
>> nealkdin
>>
>> _______________________________________________
>> sakai-dev mailing list
>> sakai-dev at collab.sakaiproject.org
>> http://collab.sakaiproject.org/mailman/listinfo/sakai-dev
>>
>> TO UNSUBSCRIBE: send email to
>> sakai-dev-unsubscribe at collab.sakaiproject.org with a subject of
>> "unsubscribe"
>>
>
>
> _______________________________________________
> sakai-dev mailing list
> sakai-dev at collab.sakaiproject.org
> http://collab.sakaiproject.org/mailman/listinfo/sakai-dev
>
> TO UNSUBSCRIBE: send email to
> sakai-dev-unsubscribe at collab.sakaiproject.org with a subject of
> "unsubscribe"
>



-- 
John Bush
602-490-0470

** This message is neither private nor confidential in fact the US
government is storing it in a warehouse located in Utah for future data
mining use cases should they arise. **
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://collab.sakaiproject.org/pipermail/sakai-dev/attachments/20140122/326c1ee4/attachment.html 
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: image/jpeg
Size: 770 bytes
Desc: not available
Url : http://collab.sakaiproject.org/pipermail/sakai-dev/attachments/20140122/326c1ee4/attachment.jpe 
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: image/jpeg
Size: 1196 bytes
Desc: not available
Url : http://collab.sakaiproject.org/pipermail/sakai-dev/attachments/20140122/326c1ee4/attachment-0001.jpe

More information about the sakai-dev mailing list