[Building Sakai] Sakai scalability monitoring and testing

[Ray Davis]

Lately at UC Berkeley we've been trying to better understand (and 
hopefully reduce) our ongoing performance and scalability woes. Some 
sakai-dev readers might be interested in the code we've recently checked 
into the Subversion repository, since it's already uncovered a couple of 
bugs and cleared up some lingering questions. None of the logic is 
particularly Berkeley-specific, and if anyone's interested in adapting 
it or further generalizing it, the task shouldn't be very difficult. But 
we're not at this moment volunteering to support the world. :)

All of scalability is divided into three parts:

A. Collecting statistics (from production and QA environments for 
diagnosis; from test environments for continuous integration testing).
B. Mimicking production activity (so that we can repeat tests and we 
aren't completely dependent on production systems for data).
C. Mimicking production data (so that we can begin tests from a known 
starting point).

For statistics, we're currently using a mix of standard Tomcat garbage 
collection logs (analyzable by gcviewer), JMX MBean statistics 
(periodically written out to the Tomcat log), and the JAMon open source 
monitoring library. JAMon's a bit stodgy, but the other options I tried 
weren't free, weren't flexible enough, or couldn't deal with Sakai's 
bizarre classloader environment. The bulk of the JMX and JAMon work is 
checked in at:

https://source.sakaiproject.org/svn/msub/berkeley.edu/bspace/jamon/sakai_2-5-x

For mimicking activity (AKA stress testing or load testing), we're 
currently using an easily configurable layer on JWebUnit / HtmlUnit. It 
doesn't do anything terribly sophisticated, but we don't need to do 
anything terribly sophisticated to bring our servers to their knees.

For mimicking production data (AKA data loading), we're using a Groovy 
script, and I'm mostly running it from the command line via the 
test-harness's component manager emulator.

(And here let me just inject an unsolicited enthusiastic plug for Thomas 
Amsler's fine, fine Sakai Groovy Shell: 
<http://confluence.sakaiproject.org/confluence/display/SGS>. Code a 
couple lines; run 'em; see what happened; edit 'em; repeat.... It's 
scarily like rapid development!)

Since cleanly repeatable stress tests need to be matched to known test 
data, both of these aspects are currently checked into the same 
top-level project:

https://source.sakaiproject.org/svn/msub/berkeley.edu/bspace/stress-test/sakai_2-5-x

The stress testers are at:

https://source.sakaiproject.org/svn/msub/berkeley.edu/stress-test/sakai_2-5-x/stresser

The data loader is at:

https://source.sakaiproject.org/svn/msub/berkeley.edu/stress-test/sakai_2-5-x/dataloader

Best,
Ray