[Building Sakai] Samigo under load exceptions
Kusnetz, Jeremy
JKusnetz at APUS.EDU
Wed Nov 16 10:32:17 PST 2011
It simply starts ramping up students, who login, navigate to a site,
select the Samigo tool, select and assessment (we have a combination of
assessments where all questions are on one page, and others where there
is one question per page) and then submit the assessment. We then reset
the site, and then repeat selecting Samigo tool, and assessment, and
submitting the assessment.
When things start to fail the CPU non idle % on the nodes are in the
60-80% utilization, about a 10 5 minute load average. JVM heap is about
at 50%. We are running 10 nodes with 5GB heaps. These are all virtual
via vmware.
The database is on physical hardware running MySQL 5.1. Right now the
database does not seem to be the bottleneck. We are approaching about
20K QPS, but CPU, load average, IO look good. The innodb_buffer_pool is
big enough to where it's almost all running in memory so no high disk
IO, plus I'm not seeing any slow queries.
Jeremy Kusnetz | Sr. Systems Engineer
American Public University System
American Military University | American Public University
661 S George Street, Charles Town, WV 25414
T 304-885-5333 | M 703-967-5212 | jkusnetz at apus.edu| www.apus.edu
<http://www.apus.edu>
From: Sam Ottenhoff [mailto:ottenhoff at longsight.com]
Sent: Wednesday, November 16, 2011 1:06 PM
To: Kusnetz, Jeremy
Cc: sakai-dev
Subject: Re: [Building Sakai] Samigo under load exceptions
Hi Jeremy,
What does your load test do?
--Sam
On Wed, Nov 16, 2011 at 12:51 PM, Kusnetz, Jeremy <JKusnetz at apus.edu>
wrote:
We are running load testing on Samigo in Sakai 2.8.0. Under load Samigo
starts failing, and almost all of the errors in the logs are the
following. Any thoughts on what this is related to?
2011-11-16 12:26:28,053 ERROR TP-Processor942
org.apache.catalina.core.ContainerBase.[Catalina].[localhost].[/samigo-a
pp].[jsp] - Servlet.service() for servlet jsp threw exception
javax.servlet.ServletException
at
com.corejsf.UploadFilter.doFilter(UploadFilter.java:161)
at
org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(Applica
tionFilterChain.java:215)
at
org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilt
erChain.java:188)
at
org.sakaiproject.util.RequestFilter.doFilter(RequestFilter.java:626)
at
org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(Applica
tionFilterChain.java:215)
at
org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilt
erChain.java:188)
at
org.apache.catalina.core.ApplicationDispatcher.invoke(ApplicationDispatc
her.java:659)
at
org.apache.catalina.core.ApplicationDispatcher.processRequest(Applicatio
nDispatcher.java:457)
at
org.apache.catalina.core.ApplicationDispatcher.doForward(ApplicationDisp
atcher.java:395)
at
org.apache.catalina.core.ApplicationDispatcher.forward(ApplicationDispat
cher.java:311)
at
com.sun.faces.context.ExternalContextImpl.dispatch(ExternalContextImpl.j
ava:322)
at
com.sun.faces.application.ViewHandlerImpl.renderView(ViewHandlerImpl.jav
a:130)
at
org.sakaiproject.jsf.app.SakaiViewHandler.renderView(SakaiViewHandler.ja
va:154)
at
com.sun.faces.lifecycle.RenderResponsePhase.execute(RenderResponsePhase.
java:87)
at
com.sun.faces.lifecycle.LifecycleImpl.phase(LifecycleImpl.java:200)
at
com.sun.faces.lifecycle.LifecycleImpl.render(LifecycleImpl.java:117)
at
javax.faces.webapp.FacesServlet.service(FacesServlet.java:198)
at
org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(Applica
tionFilterChain.java:269)
at
org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilt
erChain.java:188)
at
com.corejsf.UploadFilter.doFilter(UploadFilter.java:139)
at
org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(Applica
tionFilterChain.java:215)
at
org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilt
erChain.java:188)
at
org.sakaiproject.util.RequestFilter.doFilter(RequestFilter.java:626)
at
org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(Applica
tionFilterChain.java:215)
at
org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilt
erChain.java:188)
at
org.apache.catalina.core.ApplicationDispatcher.invoke(ApplicationDispatc
her.java:659)
at
org.apache.catalina.core.ApplicationDispatcher.processRequest(Applicatio
nDispatcher.java:457)
at
org.apache.catalina.core.ApplicationDispatcher.doForward(ApplicationDisp
atcher.java:395)
at
org.apache.catalina.core.ApplicationDispatcher.forward(ApplicationDispat
cher.java:311)
at
org.sakaiproject.jsf.util.SamigoJsfTool.dispatch(SamigoJsfTool.java:301)
at
org.sakaiproject.jsf.util.JsfTool.doPost(JsfTool.java:256)
at
javax.servlet.http.HttpServlet.service(HttpServlet.java:647)
at
javax.servlet.http.HttpServlet.service(HttpServlet.java:729)
at
org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(Applica
tionFilterChain.java:269)
at
org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilt
erChain.java:188)
at
org.apache.catalina.core.ApplicationDispatcher.invoke(ApplicationDispatc
her.java:659)
at
org.apache.catalina.core.ApplicationDispatcher.processRequest(Applicatio
nDispatcher.java:457)
at
org.apache.catalina.core.ApplicationDispatcher.doForward(ApplicationDisp
atcher.java:359)
at
org.apache.catalina.core.ApplicationDispatcher.forward(ApplicationDispat
cher.java:311)
at
org.sakaiproject.tool.impl.ActiveToolComponent$MyActiveTool.forward(Acti
veToolComponent.java:511)
at
org.sakaiproject.portal.charon.SkinnableCharonPortal.forwardTool(Skinnab
leCharonPortal.java:1467)
at
org.sakaiproject.portal.charon.handlers.ToolHandler.doTool(ToolHandler.j
ava:204)
at
org.sakaiproject.portal.charon.handlers.ToolHandler.doGet(ToolHandler.ja
va:96)
at
org.sakaiproject.portal.charon.handlers.ToolHandler.doPost(ToolHandler.j
ava:73)
at
org.sakaiproject.portal.charon.SkinnableCharonPortal.doPost(SkinnableCha
ronPortal.java:1262)
at
javax.servlet.http.HttpServlet.service(HttpServlet.java:647)
at
javax.servlet.http.HttpServlet.service(HttpServlet.java:729)
at
org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(Applica
tionFilterChain.java:269)
at
org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilt
erChain.java:188)
at
org.sakaiproject.util.RequestFilter.doFilter(RequestFilter.java:688)
at
com.rsmart.sakai.kernel.request.BrandRequestFilter.doFilter(BrandRequest
Filter.java:79)
at
org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(Applica
tionFilterChain.java:215)
at
org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilt
erChain.java:188)
at
org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValv
e.java:213)
at
org.apache.catalina.core.StandardContextValve.invoke(StandardContextValv
e.java:172)
at
org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java
:127)
at
org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java
:117)
at
org.apache.catalina.valves.AccessLogValve.invoke(AccessLogValve.java:581
)
at
org.apache.catalina.connector.RemoteIpValve.invoke(RemoteIpValve.java:60
9)
at
org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.
java:108)
at
org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:1
74)
at
org.apache.jk.server.JkCoyoteHandler.invoke(JkCoyoteHandler.java:200)
at
org.apache.jk.common.HandlerRequest.invoke(HandlerRequest.java:291)
at
org.apache.jk.common.ChannelSocket.invoke(ChannelSocket.java:775)
at
org.apache.jk.common.ChannelSocket.processConnection(ChannelSocket.java:
704)
at
org.apache.jk.common.ChannelSocket$SocketConnection.runIt(ChannelSocket.
java:897)
at
org.apache.tomcat.util.threads.ThreadPool$ControlRunnable.run(ThreadPool
.java:689)
at java.lang.Thread.run(Thread.java:662)
Caused by: org.apache.commons.fileupload.FileUploadException
at
org.apache.commons.fileupload.FileUploadBase.parseRequest(FileUploadBase
.java:385)
at
org.apache.commons.fileupload.FileUploadBase.parseRequest(FileUploadBase
.java:310)
at
com.corejsf.UploadFilter.doFilter(UploadFilter.java:125)
... 67 more
Caused by: java.io.IOException
at
org.apache.jk.common.JkInputStream.receive(JkInputStream.java:205)
at
org.apache.jk.common.JkInputStream.refillReadBuffer(JkInputStream.java:2
65)
at
org.apache.jk.common.JkInputStream.doRead(JkInputStream.java:183)
at org.apache.coyote.Request.doRead(Request.java:419)
at
org.apache.catalina.connector.InputBuffer.realReadBytes(InputBuffer.java
:270)
at
org.apache.tomcat.util.buf.ByteChunk.substract(ByteChunk.java:403)
at
org.apache.catalina.connector.InputBuffer.read(InputBuffer.java:293)
at
org.apache.catalina.connector.CoyoteInputStream.read(CoyoteInputStream.j
ava:193)
at
org.apache.commons.fileupload.MultipartStream$ItemInputStream.makeAvaila
ble(MultipartStream.java:976)
at
org.apache.commons.fileupload.MultipartStream$ItemInputStream.read(Multi
partStream.java:886)
at java.io.InputStream.read(InputStream.java:85)
at
org.apache.commons.fileupload.util.Streams.copy(Streams.java:96)
at
org.apache.commons.fileupload.util.Streams.copy(Streams.java:66)
at
org.apache.commons.fileupload.MultipartStream.readBodyData(MultipartStre
am.java:592)
at
org.apache.commons.fileupload.MultipartStream.discardBodyData(MultipartS
tream.java:618)
at
org.apache.commons.fileupload.MultipartStream.skipPreamble(MultipartStre
am.java:637)
at
org.apache.commons.fileupload.FileUploadBase$FileItemIteratorImpl.findNe
xtItem(FileUploadBase.java:984)
at
org.apache.commons.fileupload.FileUploadBase$FileItemIteratorImpl.<init>
(FileUploadBase.java:965)
at
org.apache.commons.fileupload.FileUploadBase.getItemIterator(FileUploadB
ase.java:331)
at
org.apache.commons.fileupload.FileUploadBase.parseRequest(FileUploadBase
.java:351)
... 69 more
Jeremy Kusnetz | Sr. Systems Engineer
American Public University System
American Military University | American Public University
661 S George Street, Charles Town, WV 25414
T 304-885-5333 | M 703-967-5212 | jkusnetz at apus.edu| www.apus.edu
<http://www.apus.edu>
This message is private and confidential. If you have received it in
error, please notify the sender and remove it from your system.
_______________________________________________
sakai-dev mailing list
sakai-dev at collab.sakaiproject.org
http://collab.sakaiproject.org/mailman/listinfo/sakai-dev
TO UNSUBSCRIBE: send email to
sakai-dev-unsubscribe at collab.sakaiproject.org with a subject of
"unsubscribe"
This message is private and confidential. If you have received it in error, please notify the sender and remove it from your system.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://collab.sakaiproject.org/pipermail/sakai-dev/attachments/20111116/c37cd685/attachment.html
More information about the sakai-dev
mailing list