[Building Sakai] Urgent: Apache - Tomcat AJP issues

Steve Swinsburg steve.swinsburg at gmail.com
Thu Jul 28 02:52:23 PDT 2011 

Thanks for the replies all, I'll pass that on the sysadmins and let you know how we get on.

regards,
Steve




On 28/07/2011, at 7:48 PM, Adrian Fish wrote:

> Have you set up the MaxClients property on the MPM worker? You have to do this as well as maxThreads in the AJP connector. If you don't indeterminate things start to happen when concurrent requests go up, especially on AJAXy applications.
> 
> http://cooking-sakai.blogspot.com/2011/02/tomcat-maxthreads-and-apache-maxclients.html
> 
> Cheers,
> Adrian.
> 
> On 28/07/2011 07:06, Steve Swinsburg wrote:
>> 
>> Hi all,
>> 
>> I'm assisting diagnose a problem with a production Sakai instance that is experiencing a lot of outages since upgrading to 2.8.0 from 2.5.0. Almost all of the time it seems like something is happening at the Apache-Tomcat AJP connection which locks things up and requires frequent reboots (already three today).
>> 
>> The stacktraces all come from org.apache.jk
>> 
>> 2011-07-28 14:15:43,824  WARN TP-Processor8 org.apache.jk.common.ChannelSocket - processCallbacks status 2
>> 2011-07-28 14:15:43,836  WARN TP-Processor23 org.apache.jk.common.ChannelSocket - processCallbacks status 2
>> 2011-07-28 14:15:43,864  WARN TP-Processor2 org.sakaiproject.content.impl.DbContentService - IOException 
>> java.io.IOException
>>  at org.apache.jk.common.JkInputStream.receive(JkInputStream.java:205)
>>  at org.apache.jk.common.JkInputStream.refillReadBuffer(JkInputStream.java:265)
>>  at org.apache.jk.common.JkInputStream.doRead(JkInputStream.java:183)
>>  at org.apache.coyote.Request.doRead(Request.java:419)
>>  at org.apache.catalina.connector.InputBuffer.realReadBytes(InputBuffer.java:270)
>>  at org.apache.tomcat.util.buf.ByteChunk.substract(ByteChunk.java:403)
>>  at org.apache.catalina.connector.InputBuffer.read(InputBuffer.java:293)
>>  at org.apache.catalina.connector.CoyoteInputStream.read(CoyoteInputStream.java:162)
>>  at org.sakaiproject.content.impl.DbContentService$DbStorage.putResourceBodyDb(DbContentService.java:2189)
>>  at org.sakaiproject.content.impl.DbContentService$DbStorage.commitResource(DbContentService.java:1745)
>>  at org.sakaiproject.content.impl.BaseContentService.commitResourceEdit(BaseContentService.java:5726)
>>  at org.sakaiproject.content.impl.BaseContentService.commitResource(BaseContentService.java:5539)
>>  at org.sakaiproject.dav.DavServlet.doPut(DavServlet.java:2779)
>>  at org.sakaiproject.dav.DavServlet.doDispatch(DavServlet.java:1139)
>>  at org.sakaiproject.dav.DavServlet.service(DavServlet.java:1066)
>>  at javax.servlet.http.HttpServlet.service(HttpServlet.java:729)
>>  at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:269)
>>  at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:188)
>>  at org.sakaiproject.util.RequestFilter.doFilter(RequestFilter.java:659)
>>  at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:215)
>>  at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:188)
>>  at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:213)
>>  at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:172)
>>  at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:563)
>>  at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:127)
>>  at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:117)
>>  at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:108)
>>  at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:174)
>>  at org.apache.jk.server.JkCoyoteHandler.invoke(JkCoyoteHandler.java:200)
>>  at org.apache.jk.common.HandlerRequest.invoke(HandlerRequest.java:291)
>>  at org.apache.jk.common.ChannelSocket.invoke(ChannelSocket.java:775)
>>  at org.apache.jk.common.ChannelSocket.processConnection(ChannelSocket.java:704)
>>  at org.apache.jk.common.ChannelSocket$SocketConnection.runIt(ChannelSocket.java:897)
>>  at org.apache.tomcat.util.threads.ThreadPool$ControlRunnable.run(ThreadPool.java:689)
>>  at java.lang.Thread.run(Thread.java:662)
>> 2011-07-28 14:15:43,929  WARN TP-Processor14 org.apache.jk.common.ChannelSocket - processCallbacks status 2
>> 2011-07-28 14:15:44,119  INFO main org.apache.catalina.core.StandardWrapper - Waiting for 3 instance(s) to be deallocated
>> 
>> 2011-07-28 14:15:44,495  WARN TP-Processor12 org.apache.jk.common.ChannelSocket - processCallbacks status 2
>> 2011-07-28 14:15:44,496  WARN TP-Processor18 org.apache.jk.core.MsgContext - Unable to send headers
>> java.net.SocketException: Broken pipe
>>  at java.net.SocketOutputStream.socketWrite0(Native Method)
>>  at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:92)
>>  at java.net.SocketOutputStream.write(SocketOutputStream.java:136)
>>  at org.apache.jk.common.ChannelSocket.send(ChannelSocket.java:538)
>>  at org.apache.jk.common.JkInputStream.appendHead(JkInputStream.java:326)
>>  at org.apache.jk.core.MsgContext.action(MsgContext.java:267)
>>  at org.apache.coyote.Response.action(Response.java:183)
>>  at org.apache.coyote.Response.sendHeaders(Response.java:379)
>>  at org.apache.catalina.connector.OutputBuffer.doFlush(OutputBuffer.java:313)
>>  at org.apache.catalina.connector.OutputBuffer.close(OutputBuffer.java:280)
>>  at org.apache.catalina.connector.Response.finishResponse(Response.java:484)
>>  at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:177)
>>  at org.apache.jk.server.JkCoyoteHandler.invoke(JkCoyoteHandler.java:200)
>>  at org.apache.jk.common.HandlerRequest.invoke(HandlerRequest.java:291)
>>  at org.apache.jk.common.ChannelSocket.invoke(ChannelSocket.java:775)
>>  at org.apache.jk.common.ChannelSocket.processConnection(ChannelSocket.java:704)
>>  at org.apache.jk.common.ChannelSocket$SocketConnection.runIt(ChannelSocket.java:897)
>>  at org.apache.tomcat.util.threads.ThreadPool$ControlRunnable.run(ThreadPool.java:689)
>>  at java.lang.Thread.run(Thread.java:662)
>> 
>> 
>> This time it was uploading a number of files in a folder (although total size only 4Mb).
>> 
>> The client has a standard Tomcat 8009 connector and then mod_proxy_ajp as per:
>> 
>> <VirtualHost *:80>
>>    ServerName  sakai.app.server
>>    ServerAlias sakai.server
>>    ServerAlias sakai.server.other.1
>>    ServerAlias sakai.server.other.2
>> 
>>    DocumentRoot        /var/www/html
>> 
>>    SetEnvIf client-ip ^NS$ REQ_FROM_LB
>>    SetEnvIf REQ_FROM_LB 1 DO_NOT_LOG
>>    SetEnvIf User-Agent nagios DO_NOT_LOG
>> 
>>    LogFormat "%{client-ip}i %l %u %t \"%r\" %>s %b \"%{Referer}i\" \"%{User-agent}i\" %v %A %D" combined_plus
>> 
>>    CustomLog "|cat >> /var/log/httpd/sakai-combined_log" combined_plus env=!DO_NOT_LOG
>>    ErrorLog "|cat >> /var/log/httpd/sakai-error_log"
>> 
>>    AcceptPathInfo on
>> 
>>    RedirectMatch permanent ^/$ https://sakai.server/portal
>> 
>>    # pass all requests through to local Tomcat on 8009 via AJP
>>    RewriteEngine   On
>>    RewriteCond     %{REQUEST_URI}  !^/$
>>    RewriteRule     ^/(.*)          ajp://localhost:8009/$1 [P,L]
>> 
>> </VirtualHost>
>> 
>> Any help much appreciated.
>> 
>> thanks,
>> Steve
>> 
>> 
>> 
>> 
>> _______________________________________________
>> sakai-dev mailing list
>> sakai-dev at collab.sakaiproject.org
>> http://collab.sakaiproject.org/mailman/listinfo/sakai-dev
>> 
>> TO UNSUBSCRIBE: send email to sakai-dev-unsubscribe at collab.sakaiproject.org with a subject of "unsubscribe"
> 
> -- 
> ==================================
> Adrian Fish
> Software Engineer
> Centre for e-Science
> Bowland Tower South C Floor
> Lancaster University
> Lancaster
> LA1 4YW
> email: a.fish at lancaster.ac.uk
> 
> http://confluence.sakaiproject.org/display/YAFT/Yaft
> http://confluence.sakaiproject.org/display/CLOG/Home
> http://confluence.sakaiproject.org/display/BBB/Home
> _______________________________________________
> sakai-dev mailing list
> sakai-dev at collab.sakaiproject.org
> http://collab.sakaiproject.org/mailman/listinfo/sakai-dev
> 
> TO UNSUBSCRIBE: send email to sakai-dev-unsubscribe at collab.sakaiproject.org with a subject of "unsubscribe"

-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://collab.sakaiproject.org/pipermail/sakai-dev/attachments/20110728/dc2072e5/attachment.html

More information about the sakai-dev mailing list