[Building Sakai] An odd behavior when adding participants under a high load

[Shoji Kajita]

Hi Sakai Dev,

We have started using our Sakai 2.9.x Pilot Server in real classes
since October 1st for 2012 second semester at Kyoto University, but we
have been struggling from an odd behavior.

During a high load period (18:30-18:55 last evening as shown in the
jconsole summary screen shot http://db.tt/y8uNXzWM) produced by almost
100 concurrent access from two physical class rooms, we saw the
following odd behavior when two instructors independently tried to add
students in their same course worksite:

 1. the first instructor added about 50 students by using "Add
    Participants" of Site Info, but no students could not see the
    course worksite in their screen. Refreshing pages by Web browser
    or re-loging in did not help. After several minutes (maybe seven
    minutes or so), most of students could see the course worksite
    suddenly.

    During the not-shown period, I (using admin account) sudo-ed a
    student who should have the right to access to the course
    worksite, and I could see the course worksite properly.

 2. After 10 minutes, the second instructor did the almost same thing
    and no students could not see for five minutes or so.

I have learned that the second instructor also had the same odd
behavior in his class during this evening class.

I really appreciate if someone gives me any suggestions or pointers
for this odd behavior.

Our Sakai 2.9.x Pilot Server is running under the following environment:

  a. RedHat 6.2 on VMWare (16GB main memory, 2 CPUs)

  b. Tomcat 7.0.29 with the following JAVA_OPTS:

     #  12GB in total
     #    Eden size (3GB) = New size (4GB) - 2 x Survivor space size (512MB)
     #    Survivor space size (512MB) = New size (4GB) / survivor ratio (8)

     export JAVA_OPTS='-server -Xms12288m -Xmx12288m -XX:PermSize=4096m -XX:MaxPermSize=4096m
     -XX:NewSize=4096m -XX:MaxNewSize=4096m -Djava.awt.headless=true -Dsun.lang.ClassLoader.allowArraySyntax=true
     -Dcom.sun.management.jmxremote -Dorg.apache.jasper.compiler.Parser.STRICT_WHITESPACE=false
     -Dorg.apache.jasper.compiler.Parser.STRICT_QUOTE_ESCAPING=false -Dhttp.agent=Sakai
     -Duser.language=ja -Duser.region=JP -Dfile.encoding=UTF-8 -XX:+DisableExplicitGC -XX:+PrintGCTimeStamps
     -XX:+PrintGCDetails -XX:+UseConcMarkSweepGC -XX:+CMSClassUnloadingEnabled -XX:+CMSParallelRemarkEnabled
     -XX:+UseParNewGC -verbose:gc -XX:+PrintClassHistogram -XX:+HeapDumpOnOutOfMemoryError -XX:+PrintReferenceGC
     -Xloggc:$CH/logs/loggc.out'

  c. Sakai 2.9.x (rev 113440) but kernel (rev 113905 with KNL-972.patch) and site-manage (rev 113912)

  d. Oracle 11g R1 on RedHat 6.2/VMWare 

Best regards,
Shoji Kajita
---
Shoji Kajita
IT Planning Office, IIMC, Kyoto University, Japan
http://about.me/shojikajita