[Building Sakai] Samigo Load Testing - details for ApereoCamp
Karen Tsao
ktsao at stanford.edu
Wed Jan 29 14:38:00 PST 2014
Hi Beth,
Thanks for sharing the test result. Here are some performance improvement
in 10:
SAM-1472: Scores page load times excessive for large sites with assessments
released to groups.
SAM-1701: Performance regression from grading attachment functionality
SAM-2063: Performance improvement for Auto Submission
It'll be great if you can repeat the test for Samigo 10.
Thanks,
Karen
On Mon, Jan 27, 2014 at 6:19 AM, Kirschner, Beth <bkirschn at umich.edu> wrote:
> Thanks Stephen - we are planning on load testing Sakai 10 in a couple
> months - I'll see if it's feasible to re-test Samigo as part of our Sakai
> 10 testing.
>
> - Beth
>
> On Jan 27, 2014, at 5:09 AM, Stephen Marquard <stephen.marquard at uct.ac.za>
> wrote:
>
> > Hi Beth
> >
> > Great to see this testing, and thanks for sharing details. There is one
> change which David Horwitz contributed to Samigo code as a result of some
> performance investigation which is:
> >
> > https://jira.sakaiproject.org/browse/SAM-1615
> > POJO'ise assesment objects
> >
> > This is in trunk / 10.x but not 2.9.x. We have seen cases where Samigo
> seemed to be issuing an unnecessary level of SQL update queries during an
> assessment with 2.9.x, and it's possible this change could fix that
> behaviour though we haven't tested it thoroughly.
> >
> > If it's possible for you to do so, I think it would be worth running
> your same load tests with trunk code and seeing if there are any
> significant differences in behaviour.
> >
> > Regards
> > Stephen
> >
> > ---
> > Stephen Marquard, Learning Technologies Co-ordinator
> > Centre for Innovation in Learning and Teaching (CILT, formerly CET and
> HAESDU)
> > University of Cape Town
> > http://www.cet.uct.ac.za
> > stephen.marquard at uct.ac.za
> > Phone: +27-21-650-5037 Cell: +27-83-500-5290
> >
> > From: sakai-dev-bounces at collab.sakaiproject.org [mailto:
> sakai-dev-bounces at collab.sakaiproject.org] On Behalf Of Kirschner, Beth
> > Sent: 25 January 2014 12:45 AM
> > To: <sakai-dev at collab.sakaiproject.org> Dev
> > Subject: [Building Sakai] Samigo Load Testing - details for ApereoCamp
> >
> > Hi everyone,
> >
> > The University of Michigan performed some load tests against Samigo
> (v2.9.2), in order to assess how it performs in some usage scenarios we had
> been considering for Samigo:
> > Test Case #1: Deliver a test to a course containing 1500
> Students
> > Test Case #2: Instructor views and releases grades to the
> Gradebook after the delivery of the tests in case #1
> > Test Case #3: Deliver several tests concurrently to several
> large course sites
> >
> > We were using our LoadRunner infrastructure, and constructed load test
> scripts for the above scenarios. We uncovered some details on performance
> problems that I think are already known within the community (see complete
> details below), and I wanted to share these details in advance of the
> upcoming ApereoCamp, where a fair share of the PMC and other developers
> will be gathered.
> >
> > Does anyone know if there have been significant changes to Samigo to
> warrant us re-testing with Sakai 10.0? If not, I'm hoping these test
> results will provide useful information for further analysis of the code.
> While we can't commit to re-running these tests again, it's something I
> would consider discussing if changes were made that might address some or
> all of these performance bottle-necks.
> >
> > We didn't execute any load tests with courses containing
> small-to-moderate class sizes, and it's our expectation that Samigo
> performs well in these usage scenarios (based on anecdotal evidence).
> >
> > Let me know if anyone has questions parsing the details below.
> >
> > Thanks!
> > - Beth
> >
> > Test Case #1: Deliver a test to a course containing 1500 Students
> > Three tests were run: in the first test scores automatically released to
> the Gradebook, the second did not release scores automatically, the third
> pulled questions from a 200 question pool. The students took a 17 question
> timed test, all starting within two minutes of one another. The rationale
> here was that a failure would indicate that Samigo would be a poor fit for
> high stakes testing of large courses.
> >
> > In general, Samigo handled this load fairly smoothly. Database server
> CPU never climbed above 10% usage and response time were generally below 4
> seconds per transaction. Two exceptions were beginning the assessment,
> which average over 9 seconds and was highly variable, and submission of
> 'Fill-in-the-blank' question types, which averaged over 26 seconds. The
> last result is surprising, as the rest of the question types were fairly
> consistent. This may mean bug exists in the code which processes this
> question type.
> >
> > Notes:
> > * Results for the three tests were similar, with the
> non-auto released test seeing higher app server CPU (81% vs 69%) than the
> auto-released test.
> > * The most expensive database query was a session query,
> with the second being a Samigo query. These two were significantly higher
> consumers than the rest:
> > * SQL ID: btjfnypk04v55
> > select
> AX.SESSION_ID,AX.SESSION_SERVER,AX.SESSION_USER,AX.SESSION_IP,AX.SESSION_HOSTNAME,AX.SESSION_USER_AGENT,AX.SESSION_START,AX.SESSION_END,AX.SESSION_ACTIVE
> from SAKAI_SESSION AX inner join SAKAI_PRESENCE A ON AX.SESSION_ID =
> A.SESSION_ID where AX.SESSION_ACTIVE=:"SYS_B_0" and A.LOCATION_ID = :1
> > * SQL ID: 22hdsavthcf22
> > select assessment0_.ASSESSMENTID as ASSESSME2_1_,
> assessment0_.ASSESSMENTMETADATAID as ASSESSME1_1_,
> assessment0_.ASSESSMENTMETADATAID as ASSESSME1_209_0_, asse
> > ssment0_.ASSESSMENTID as ASSESSME2_209_0_, assessment0_.LABEL as
> LABEL209_0_, assessment0_.ENTRY as ENTRY209_0_ from SAM_PUBLISHEDMETADATA_T
> assessment0_ where
> > * Additionally, the question pool test saw high resource
> consumption from the following SQL:
> > * SQL ID: 055qhz3wab4u7
> > select count(assessment0_.ASSESSMENTGRADINGID) as col_0_0_ from
> SAM_ASSESSMENTGRADING_T assessment0_ where assessment0_.FORGRADE=:1 and
> assessment0_.AGENTID=:2 and assessment0_.PUBLISHEDASSESSMENTID=:3
> > * SQL ID: cugj6729dx5gu
> > select assessment0_.ASSESSMENTGRADINGID as ASSESSME1_221_,
> assessment0_.PUBLISHEDASSESSMENTID as PUBLISHE2_221_, assessment0_.AGENTID
> as AGENTID221_, assessment0_.SUBMITTEDDATE as SUBMITTE4_221_,
> assessment0_.ISLATE as ISLATE221_, assessment0_.FORGRADE as FORGRADE221_,
> assessment0_.TOTALAUTOSCORE as TOTALAUT7_221_, ass
> > * SQL ID: bzhgbh30kq1xy
> > select itemgradin0_.ITEMGRADINGID as ITEMGRAD1_220_,
> itemgradin0_.ASSESSMENTGRADINGID as ASSESSME2_220_,
> itemgradin0_.PUBLISHEDITEMID as PUBLISHE3_220_,
> itemgradin0_.PUBLISHEDITEMTEXTID as PUBLISHE4_220_, itemgradin0_.AGENTID as
> AGENTID220_, itemgradin0_.SUBMITTEDDATE as SUBMITTE6_220_,
> itemgradin0_.PUBLISHEDANSWERID a
> > * SQL ID: fmx20pg0zdaf5
> > select answerset0_.ITEMTEXTID as ITEMTEXTID7_, answerset0_.ANSWERID as
> ANSWERID7_, answerset0_.ANSWERID as ANSWERID217_6_, answerset0_.ITEMTEXTID
> as ITEMTEXTID217_6_, answerset0_.ITEMID as ITEMID217_6_, answerset0_.TEXT
> as TEXT217_6_, answerset0_.SEQUENCE as SEQUENCE217_6_, answerset0_.LABEL as
> LABEL217_6_, answerset0_
> > * The application tier never utilized more than 10
> connection at any time on any of the 7 application servers. This indicates
> that the database communicated efficiently throughout the test.
> >
> >
> > Test Case #2: Instructor views and releases grades to the Gradebook
> after the delivery of the tests in case #1
> > This case focused on the instructor viewing the results from the three
> tests above. Specifically, three courses of 1500 students, one with grades
> automatically released to the gradebook, the second and third where grades
> were not previously released. The third course contained an assessment
> pulled from a question pool In the case where grades were not
> automatically released, the instructor released the grades after viewing
> them. Note, these were single user tests.
> >
> > Opening the initial list of students in all three courses is lengthy
> (from 36 - 51 seconds). The default view is of 200 students. The tests
> then attempted to open the full list of 1500 students on a single page.
> This transaction took 140 seconds in the course where the grades had been
> automatically released, 78 seconds where the grades were not automatically
> released, and--strangely--15 seconds in the course where the assessment
> pulled from a question pool and grades were not automatically released to
> the gradebook. One conclusion is that the connection between Samigo and
> the Gradebook is an expensive one for display purposes.
> >
> > The disparity in viewing the full list of students for the non-released
> assessments is curious. No conclusions can be drawn from the database
> usage report. In both cases, Database CPU is the primary contributor to
> usage time, and the same queries populate the top of the usage report. In
> the case where the questions were pulled from a question pool, the queries
> were executed 25% fewer times than when the assessment did not use a
> question pool. Here are the top resource consumers from a SQL perspective
> in both cases:
> > * SQL ID: 22hdsavthcf22
> > select assessment0_.ASSESSMENTID as ASSESSME2_1_,
> assessment0_.ASSESSMENTMETADATAID as ASSESSME1_1_,
> assessment0_.ASSESSMENTMETADATAID as ASSESSME1_209_0_,
> assessment0_.ASSESSMENTID as ASSESSME2_209_0_, assessment0_.LABEL as
> LABEL209_0_, assessment0_.ENTRY as ENTRY209_0_ from SAM_PUBLISHEDMETADATA_T
> assessment0_ where
> > * SQL ID: 8ayxw2af9nc75
> > select assessment0_.ASSESSMENTID as ASSESSM15_1_,
> assessment0_.ATTACHMENTID as ATTACHME1_1_, assessment0_.ATTACHMENTID as
> ATTACHME1_219_0_, assessment0_.RESOURC
> > EID as RESOURCEID219_0_, assessment0_.FILENAME as FILENAME219_0_,
> assessment0_.MIMETYPE as MIMETYPE219_0_, assessment0_.FILESIZE as
> FILESIZE219_0_, assessment0_
> >
> > In both cases, the next four resource consuming SQLs were not
> Samigo-related.
> >
> > Releasing the grades to the gradebook took 166 seconds both for the
> question pool and non-question pool assessments.
> >
> > Test Case #3: Deliver several tests concurrently to several large course
> sites
> > This was a stress test designed to drive the system to the point of
> failure. We were interested in finding out what component would be the
> primary bottleneck--and why--in the case of high Samigo stress.
> >
> > Application server CPU usage bounced around 100% during this test, while
> database CPU usage remained in the 22-32% range. Database pool connections
> occasionally spiked up to 50 per server, a very high number for our
> configuration. While this is normally indicative of slow database
> response, it was not consistent. Servers frequently returned from their
> spikes to levels below 5. Database connection pool spikes can also be
> caused by high CPU usage on the application server, as the connections are
> not cleaned in a timely fashion as the app server struggles to keep up with
> load. The stress pattern in this test is more consistent with this
> explanation.
> >
> > Note: the application server was a bottleneck in our configuration. But
> this may not mean it will be the primary bottleneck in other
> configurations. Our database has 32 physical cores. Other machines
> without that type of CPU pool may experience problematic conditions at
> lower levels of load.
> >
> > UNIVERSITY OF CAPE TOWN
> >
> > This e-mail is subject to the UCT ICT policies and e-mail disclaimer
> published on our website at
> http://www.uct.ac.za/about/policies/emaildisclaimer/ or obtainable from +27
> 21 650 9111. This e-mail is intended only for the person(s) to whom it is
> addressed. If the e-mail has reached you in error, please notify the
> author. If you are not the intended recipient of the e-mail you may not
> use, disclose, copy, redirect or print the content. If this e-mail is not
> related to the business of UCT it is sent by the sender in the sender's
> individual capacity.
>
> _______________________________________________
> sakai-dev mailing list
> sakai-dev at collab.sakaiproject.org
> http://collab.sakaiproject.org/mailman/listinfo/sakai-dev
>
> TO UNSUBSCRIBE: send email to
> sakai-dev-unsubscribe at collab.sakaiproject.org with a subject of
> "unsubscribe"
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://collab.sakaiproject.org/pipermail/sakai-dev/attachments/20140129/b5966ad2/attachment.html
More information about the sakai-dev
mailing list