[Building Sakai] Urgent: Apache - Tomcat AJP issues
Adrian Fish
a.fish at lancaster.ac.uk
Thu Jul 28 02:48:48 PDT 2011
Have you set up the MaxClients property on the MPM worker? You have to
do this as well as maxThreads in the AJP connector. If you don't
indeterminate things start to happen when concurrent requests go up,
especially on AJAXy applications.
http://cooking-sakai.blogspot.com/2011/02/tomcat-maxthreads-and-apache-maxclients.html
Cheers,
Adrian.
On 28/07/2011 07:06, Steve Swinsburg wrote:
> Hi all,
>
> I'm assisting diagnose a problem with a production Sakai instance that
> is experiencing a lot of outages since upgrading to 2.8.0 from 2.5.0.
> Almost all of the time it seems like something is happening at the
> Apache-Tomcat AJP connection which locks things up and requires
> frequent reboots (already three today).
>
> The stacktraces all come from org.apache.jk
>
> 2011-07-28 14:15:43,824 WARN TP-Processor8
> org.apache.jk.common.ChannelSocket - processCallbacks status 2
> 2011-07-28 14:15:43,836 WARN TP-Processor23
> org.apache.jk.common.ChannelSocket - processCallbacks status 2
> 2011-07-28 14:15:43,864 WARN TP-Processor2
> org.sakaiproject.content.impl.DbContentService - IOException
> java.io.IOException
> at org.apache.jk.common.JkInputStream.receive(JkInputStream.java:205)
> at
> org.apache.jk.common.JkInputStream.refillReadBuffer(JkInputStream.java:265)
> at org.apache.jk.common.JkInputStream.doRead(JkInputStream.java:183)
> at org.apache.coyote.Request.doRead(Request.java:419)
> at
> org.apache.catalina.connector.InputBuffer.realReadBytes(InputBuffer.java:270)
> at org.apache.tomcat.util.buf.ByteChunk.substract(ByteChunk.java:403)
> at org.apache.catalina.connector.InputBuffer.read(InputBuffer.java:293)
> at
> org.apache.catalina.connector.CoyoteInputStream.read(CoyoteInputStream.java:162)
> at
> org.sakaiproject.content.impl.DbContentService$DbStorage.putResourceBodyDb(DbContentService.java:2189)
> at
> org.sakaiproject.content.impl.DbContentService$DbStorage.commitResource(DbContentService.java:1745)
> at
> org.sakaiproject.content.impl.BaseContentService.commitResourceEdit(BaseContentService.java:5726)
> at
> org.sakaiproject.content.impl.BaseContentService.commitResource(BaseContentService.java:5539)
> at org.sakaiproject.dav.DavServlet.doPut(DavServlet.java:2779)
> at org.sakaiproject.dav.DavServlet.doDispatch(DavServlet.java:1139)
> at org.sakaiproject.dav.DavServlet.service(DavServlet.java:1066)
> at javax.servlet.http.HttpServlet.service(HttpServlet.java:729)
> at
> org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:269)
> at
> org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:188)
> at org.sakaiproject.util.RequestFilter.doFilter(RequestFilter.java:659)
> at
> org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:215)
> at
> org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:188)
> at
> org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:213)
> at
> org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:172)
> at
> org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:563)
> at
> org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:127)
> at
> org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:117)
> at
> org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:108)
> at
> org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:174)
> at org.apache.jk.server.JkCoyoteHandler.invoke(JkCoyoteHandler.java:200)
> at org.apache.jk.common.HandlerRequest.invoke(HandlerRequest.java:291)
> at org.apache.jk.common.ChannelSocket.invoke(ChannelSocket.java:775)
> at
> org.apache.jk.common.ChannelSocket.processConnection(ChannelSocket.java:704)
> at
> org.apache.jk.common.ChannelSocket$SocketConnection.runIt(ChannelSocket.java:897)
> at
> org.apache.tomcat.util.threads.ThreadPool$ControlRunnable.run(ThreadPool.java:689)
> at java.lang.Thread.run(Thread.java:662)
> 2011-07-28 14:15:43,929 WARN TP-Processor14
> org.apache.jk.common.ChannelSocket - processCallbacks status 2
> 2011-07-28 14:15:44,119 INFO main
> org.apache.catalina.core.StandardWrapper - Waiting for 3 instance(s)
> to be deallocated
>
> 2011-07-28 14:15:44,495 WARN TP-Processor12
> org.apache.jk.common.ChannelSocket - processCallbacks status 2
> 2011-07-28 14:15:44,496 WARN TP-Processor18
> org.apache.jk.core.MsgContext - Unable to send headers
> java.net.SocketException: Broken pipe
> at java.net.SocketOutputStream.socketWrite0(Native Method)
> at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:92)
> at java.net.SocketOutputStream.write(SocketOutputStream.java:136)
> at org.apache.jk.common.ChannelSocket.send(ChannelSocket.java:538)
> at org.apache.jk.common.JkInputStream.appendHead(JkInputStream.java:326)
> at org.apache.jk.core.MsgContext.action(MsgContext.java:267)
> at org.apache.coyote.Response.action(Response.java:183)
> at org.apache.coyote.Response.sendHeaders(Response.java:379)
> at
> org.apache.catalina.connector.OutputBuffer.doFlush(OutputBuffer.java:313)
> at org.apache.catalina.connector.OutputBuffer.close(OutputBuffer.java:280)
> at
> org.apache.catalina.connector.Response.finishResponse(Response.java:484)
> at
> org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:177)
> at org.apache.jk.server.JkCoyoteHandler.invoke(JkCoyoteHandler.java:200)
> at org.apache.jk.common.HandlerRequest.invoke(HandlerRequest.java:291)
> at org.apache.jk.common.ChannelSocket.invoke(ChannelSocket.java:775)
> at
> org.apache.jk.common.ChannelSocket.processConnection(ChannelSocket.java:704)
> at
> org.apache.jk.common.ChannelSocket$SocketConnection.runIt(ChannelSocket.java:897)
> at
> org.apache.tomcat.util.threads.ThreadPool$ControlRunnable.run(ThreadPool.java:689)
> at java.lang.Thread.run(Thread.java:662)
>
>
> This time it was uploading a number of files in a folder (although
> total size only 4Mb).
>
> The client has a standard Tomcat 8009 connector and then mod_proxy_ajp
> as per:
>
> <VirtualHost *:80>
> ServerName sakai.app.server
> ServerAlias sakai.server
> ServerAlias sakai.server.other.1
> ServerAlias sakai.server.other.2
>
> DocumentRoot /var/www/html
>
> SetEnvIf client-ip ^NS$ REQ_FROM_LB
> SetEnvIf REQ_FROM_LB 1 DO_NOT_LOG
> SetEnvIf User-Agent nagios DO_NOT_LOG
>
> LogFormat "%{client-ip}i %l %u %t \"%r\" %>s %b \"%{Referer}i\"
> \"%{User-agent}i\" %v %A %D" combined_plus
>
> CustomLog "|cat >> /var/log/httpd/sakai-combined_log" combined_plus
> env=!DO_NOT_LOG
> ErrorLog "|cat >> /var/log/httpd/sakai-error_log"
>
> AcceptPathInfo on
>
> RedirectMatch permanent ^/$ https://sakai.server/portal
>
> # pass all requests through to local Tomcat on 8009 via AJP
> RewriteEngine On
> RewriteCond %{REQUEST_URI} !^/$
> RewriteRule ^/(.*) ajp://localhost:8009/$1 [P,L]
>
> </VirtualHost>
>
> Any help much appreciated.
>
> thanks,
> Steve
>
>
>
>
> _______________________________________________
> sakai-dev mailing list
> sakai-dev at collab.sakaiproject.org
> http://collab.sakaiproject.org/mailman/listinfo/sakai-dev
>
> TO UNSUBSCRIBE: send email to sakai-dev-unsubscribe at collab.sakaiproject.org with a subject of "unsubscribe"
--
==================================
Adrian Fish
Software Engineer
Centre for e-Science
Bowland Tower South C Floor
Lancaster University
Lancaster
LA1 4YW
email: a.fish at lancaster.ac.uk
http://confluence.sakaiproject.org/display/YAFT/Yaft
http://confluence.sakaiproject.org/display/CLOG/Home
http://confluence.sakaiproject.org/display/BBB/Home
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://collab.sakaiproject.org/pipermail/sakai-dev/attachments/20110728/7dfde9e0/attachment.html
More information about the sakai-dev
mailing list