[Building Sakai] Samigo Load Testing - details for ApereoCamp

Fri Jan 24 14:44:45 PST 2014

Hi everyone,

The University of Michigan performed some load tests against Samigo (v2.9.2), in order to assess how it performs in some usage scenarios we had been considering for Samigo:
	Test Case #1: Deliver a test to a course containing 1500 Students
	Test Case #2: Instructor views and releases grades to the Gradebook after the delivery of the tests in case #1
	Test Case #3: Deliver several tests concurrently to several large course sites

We were using our LoadRunner infrastructure, and constructed load test scripts for the above scenarios. We uncovered some details on performance problems that I think are already known within the community (see complete details below), and I wanted to share these details in advance of the upcoming ApereoCamp, where a fair share of the PMC and other developers will be gathered.

Does anyone know if there have been significant changes to Samigo to warrant us re-testing with Sakai 10.0? If not, I'm hoping these test results will provide useful information for further analysis of the code. While we can't commit to re-running these tests again, it's something I would consider discussing if changes were made that might address some or all of these performance bottle-necks.

We didn't execute any load tests with courses containing small-to-moderate class sizes, and it's our expectation that Samigo performs well in these usage scenarios (based on anecdotal evidence).

Let me know if anyone has questions parsing the details below.

Thanks!
- Beth

Test Case #1: Deliver a test to a course containing 1500 Students
Three tests were run: in the first test scores automatically released to the Gradebook, the second did not release scores automatically, the third pulled questions from a 200 question pool.  The students took a 17 question timed test, all starting within two minutes of one another.  The rationale here was that a failure would indicate that Samigo would be a poor fit for high stakes testing of large courses.

In general, Samigo handled this load fairly smoothly.  Database server CPU never climbed above 10% usage and response time were generally below 4 seconds per transaction.  Two exceptions were beginning the assessment, which average over 9 seconds and was highly variable, and submission of ‘Fill-in-the-blank’ question types, which averaged over 26 seconds.  The last result is surprising, as the rest of the question types were fairly consistent.  This may mean bug exists in the code which processes this question type.

Notes:
	• Results for the three tests were similar, with the non-auto released test seeing higher app server CPU (81% vs 69%) than the auto-released test.
	• The most expensive database query was a session query, with the second being a Samigo query.  These two were significantly higher consumers than the rest:
		• SQL ID: btjfnypk04v55
select AX.SESSION_ID,AX.SESSION_SERVER,AX.SESSION_USER,AX.SESSION_IP,AX.SESSION_HOSTNAME,AX.SESSION_USER_AGENT,AX.SESSION_START,AX.SESSION_END,AX.SESSION_ACTIVE  from SAKAI_SESSION AX inner join SAKAI_PRESENCE A ON AX.SESSION_ID = A.SESSION_ID where AX.SESSION_ACTIVE=:"SYS_B_0" and A.LOCATION_ID = :1
		• SQL ID: 22hdsavthcf22
select assessment0_.ASSESSMENTID as ASSESSME2_1_, assessment0_.ASSESSMENTMETADATAID as ASSESSME1_1_, assessment0_.ASSESSMENTMETADATAID as ASSESSME1_209_0_, asse
ssment0_.ASSESSMENTID as ASSESSME2_209_0_, assessment0_.LABEL as LABEL209_0_, assessment0_.ENTRY as ENTRY209_0_ from SAM_PUBLISHEDMETADATA_T assessment0_ where
	• Additionally, the question pool test saw high resource consumption from the following SQL:
		• SQL ID: 055qhz3wab4u7 
select count(assessment0_.ASSESSMENTGRADINGID) as col_0_0_ from SAM_ASSESSMENTGRADING_T assessment0_ where assessment0_.FORGRADE=:1 and assessment0_.AGENTID=:2  and assessment0_.PUBLISHEDASSESSMENTID=:3
	• SQL ID: cugj6729dx5gu
select assessment0_.ASSESSMENTGRADINGID as ASSESSME1_221_, assessment0_.PUBLISHEDASSESSMENTID as PUBLISHE2_221_, assessment0_.AGENTID as AGENTID221_, assessment0_.SUBMITTEDDATE as SUBMITTE4_221_, assessment0_.ISLATE as ISLATE221_, assessment0_.FORGRADE as FORGRADE221_, assessment0_.TOTALAUTOSCORE as TOTALAUT7_221_, ass
	• SQL ID: bzhgbh30kq1xy
select itemgradin0_.ITEMGRADINGID as ITEMGRAD1_220_, itemgradin0_.ASSESSMENTGRADINGID as ASSESSME2_220_, itemgradin0_.PUBLISHEDITEMID as PUBLISHE3_220_, itemgradin0_.PUBLISHEDITEMTEXTID as PUBLISHE4_220_, itemgradin0_.AGENTID as AGENTID220_, itemgradin0_.SUBMITTEDDATE as SUBMITTE6_220_, itemgradin0_.PUBLISHEDANSWERID a
	• SQL ID: fmx20pg0zdaf5
select answerset0_.ITEMTEXTID as ITEMTEXTID7_, answerset0_.ANSWERID as ANSWERID7_, answerset0_.ANSWERID as ANSWERID217_6_, answerset0_.ITEMTEXTID as ITEMTEXTID217_6_, answerset0_.ITEMID as ITEMID217_6_, answerset0_.TEXT as TEXT217_6_, answerset0_.SEQUENCE as SEQUENCE217_6_, answerset0_.LABEL as LABEL217_6_, answerset0_
	• The application tier never utilized more than 10 connection at any time on any of the 7 application servers.  This indicates that the database communicated efficiently throughout the test.

Test Case #2: Instructor views and releases grades to the Gradebook after the delivery of the tests in case #1
This case focused on the instructor viewing the results from the three tests above.  Specifically, three courses of 1500 students, one with grades automatically released to the gradebook, the second and third where grades were not previously released.  The third course contained an assessment pulled from a question pool  In the case where grades were not automatically released, the instructor released the grades after viewing them.  Note, these were single user tests.

Opening the initial list of students in all three courses is lengthy (from 36 - 51 seconds).  The default view is of 200 students.  The tests then attempted to open the full list of 1500 students on a single page.  This transaction took 140 seconds in the course where the grades had been automatically released, 78 seconds where the grades were not automatically released, and--strangely--15 seconds in the course where the assessment pulled from a question pool and grades were not automatically released to the gradebook.  One conclusion is that the connection between Samigo and the Gradebook is an expensive one for display purposes.

The disparity in viewing the full list of students for the non-released assessments is curious.  No conclusions can be drawn from the database usage report.  In both cases, Database CPU is the primary contributor to usage time, and the same queries populate the top of the usage report.  In the case where the questions were pulled from a question pool, the queries were executed 25% fewer times than when the assessment did not use a question pool.  Here are the top resource consumers from a SQL perspective in both cases:
	• SQL ID: 22hdsavthcf22
select assessment0_.ASSESSMENTID as ASSESSME2_1_, assessment0_.ASSESSMENTMETADATAID as ASSESSME1_1_, assessment0_.ASSESSMENTMETADATAID as ASSESSME1_209_0_, assessment0_.ASSESSMENTID as ASSESSME2_209_0_, assessment0_.LABEL as LABEL209_0_, assessment0_.ENTRY as ENTRY209_0_ from SAM_PUBLISHEDMETADATA_T assessment0_ where
	• SQL ID: 8ayxw2af9nc75
select assessment0_.ASSESSMENTID as ASSESSM15_1_, assessment0_.ATTACHMENTID as ATTACHME1_1_, assessment0_.ATTACHMENTID as ATTACHME1_219_0_, assessment0_.RESOURC
EID as RESOURCEID219_0_, assessment0_.FILENAME as FILENAME219_0_, assessment0_.MIMETYPE as MIMETYPE219_0_, assessment0_.FILESIZE as FILESIZE219_0_, assessment0_

In both cases, the next four resource consuming SQLs were not Samigo-related.

Releasing the grades to the gradebook took 166 seconds both for the question pool and non-question pool assessments.

Test Case #3: Deliver several tests concurrently to several large course sites
This was a stress test designed to drive the system to the point of failure.  We were interested in finding out what component would be the primary bottleneck--and why--in the case of high Samigo stress.

Application server CPU usage bounced around 100% during this test, while database CPU usage remained in the 22-32% range.  Database pool connections occasionally spiked up to 50 per server, a very high number for our configuration.  While this is normally indicative of slow database response, it was not consistent.  Servers frequently returned from their spikes to levels below 5.  Database connection pool spikes can also be caused by high CPU usage on the application server, as the connections are not cleaned in a timely fashion as the app server struggles to keep up with load.  The stress pattern in this test is more consistent with this explanation.

Note: the application server was a bottleneck in our configuration.  But this may not mean it will be the primary bottleneck in other configurations.  Our database has 32 physical cores.  Other machines without that type of CPU pool may experience problematic conditions at lower levels of load.

-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://collab.sakaiproject.org/pipermail/sakai-dev/attachments/20140124/fd97e8d2/attachment.html 