[Deploying Sakai] Urgent: Apache - Tomcat AJP issues
Steve Swinsburg
steve.swinsburg at gmail.com
Wed Jul 27 23:18:26 PDT 2011
Also, around the time of the latest outage I see a lot of these error messages in the Apache error_log
[Thu Jul 28 14:15:16 2011] [error] (70007)The timeout specified has expired: ajp_ilink_receive() can't receive header
[Thu Jul 28 14:15:20 2011] [error] (70007)The timeout specified has expired: ajp_ilink_receive() can't receive header
[Thu Jul 28 14:15:54 2011] [error] (104)Connection reset by peer: ajp_ilink_receive() can't receive header
Thanks,
Steve
On 28/07/2011, at 4:06 PM, Steve Swinsburg wrote:
> Hi all,
>
> I'm assisting diagnose a problem with a production Sakai instance that is experiencing a lot of outages since upgrading to 2.8.0 from 2.5.0. Almost all of the time it seems like something is happening at the Apache-Tomcat AJP connection which locks things up and requires frequent reboots (already three today).
>
> The stacktraces all come from org.apache.jk
>
> 2011-07-28 14:15:43,824 WARN TP-Processor8 org.apache.jk.common.ChannelSocket - processCallbacks status 2
> 2011-07-28 14:15:43,836 WARN TP-Processor23 org.apache.jk.common.ChannelSocket - processCallbacks status 2
> 2011-07-28 14:15:43,864 WARN TP-Processor2 org.sakaiproject.content.impl.DbContentService - IOException
> java.io.IOException
> at org.apache.jk.common.JkInputStream.receive(JkInputStream.java:205)
> at org.apache.jk.common.JkInputStream.refillReadBuffer(JkInputStream.java:265)
> at org.apache.jk.common.JkInputStream.doRead(JkInputStream.java:183)
> at org.apache.coyote.Request.doRead(Request.java:419)
> at org.apache.catalina.connector.InputBuffer.realReadBytes(InputBuffer.java:270)
> at org.apache.tomcat.util.buf.ByteChunk.substract(ByteChunk.java:403)
> at org.apache.catalina.connector.InputBuffer.read(InputBuffer.java:293)
> at org.apache.catalina.connector.CoyoteInputStream.read(CoyoteInputStream.java:162)
> at org.sakaiproject.content.impl.DbContentService$DbStorage.putResourceBodyDb(DbContentService.java:2189)
> at org.sakaiproject.content.impl.DbContentService$DbStorage.commitResource(DbContentService.java:1745)
> at org.sakaiproject.content.impl.BaseContentService.commitResourceEdit(BaseContentService.java:5726)
> at org.sakaiproject.content.impl.BaseContentService.commitResource(BaseContentService.java:5539)
> at org.sakaiproject.dav.DavServlet.doPut(DavServlet.java:2779)
> at org.sakaiproject.dav.DavServlet.doDispatch(DavServlet.java:1139)
> at org.sakaiproject.dav.DavServlet.service(DavServlet.java:1066)
> at javax.servlet.http.HttpServlet.service(HttpServlet.java:729)
> at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:269)
> at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:188)
> at org.sakaiproject.util.RequestFilter.doFilter(RequestFilter.java:659)
> at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:215)
> at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:188)
> at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:213)
> at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:172)
> at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:563)
> at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:127)
> at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:117)
> at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:108)
> at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:174)
> at org.apache.jk.server.JkCoyoteHandler.invoke(JkCoyoteHandler.java:200)
> at org.apache.jk.common.HandlerRequest.invoke(HandlerRequest.java:291)
> at org.apache.jk.common.ChannelSocket.invoke(ChannelSocket.java:775)
> at org.apache.jk.common.ChannelSocket.processConnection(ChannelSocket.java:704)
> at org.apache.jk.common.ChannelSocket$SocketConnection.runIt(ChannelSocket.java:897)
> at org.apache.tomcat.util.threads.ThreadPool$ControlRunnable.run(ThreadPool.java:689)
> at java.lang.Thread.run(Thread.java:662)
> 2011-07-28 14:15:43,929 WARN TP-Processor14 org.apache.jk.common.ChannelSocket - processCallbacks status 2
> 2011-07-28 14:15:44,119 INFO main org.apache.catalina.core.StandardWrapper - Waiting for 3 instance(s) to be deallocated
>
> 2011-07-28 14:15:44,495 WARN TP-Processor12 org.apache.jk.common.ChannelSocket - processCallbacks status 2
> 2011-07-28 14:15:44,496 WARN TP-Processor18 org.apache.jk.core.MsgContext - Unable to send headers
> java.net.SocketException: Broken pipe
> at java.net.SocketOutputStream.socketWrite0(Native Method)
> at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:92)
> at java.net.SocketOutputStream.write(SocketOutputStream.java:136)
> at org.apache.jk.common.ChannelSocket.send(ChannelSocket.java:538)
> at org.apache.jk.common.JkInputStream.appendHead(JkInputStream.java:326)
> at org.apache.jk.core.MsgContext.action(MsgContext.java:267)
> at org.apache.coyote.Response.action(Response.java:183)
> at org.apache.coyote.Response.sendHeaders(Response.java:379)
> at org.apache.catalina.connector.OutputBuffer.doFlush(OutputBuffer.java:313)
> at org.apache.catalina.connector.OutputBuffer.close(OutputBuffer.java:280)
> at org.apache.catalina.connector.Response.finishResponse(Response.java:484)
> at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:177)
> at org.apache.jk.server.JkCoyoteHandler.invoke(JkCoyoteHandler.java:200)
> at org.apache.jk.common.HandlerRequest.invoke(HandlerRequest.java:291)
> at org.apache.jk.common.ChannelSocket.invoke(ChannelSocket.java:775)
> at org.apache.jk.common.ChannelSocket.processConnection(ChannelSocket.java:704)
> at org.apache.jk.common.ChannelSocket$SocketConnection.runIt(ChannelSocket.java:897)
> at org.apache.tomcat.util.threads.ThreadPool$ControlRunnable.run(ThreadPool.java:689)
> at java.lang.Thread.run(Thread.java:662)
>
>
> This time it was uploading a number of files in a folder (although total size only 4Mb).
>
> The client has a standard Tomcat 8009 connector and then mod_proxy_ajp as per:
>
> <VirtualHost *:80>
> ServerName sakai.app.server
> ServerAlias sakai.server
> ServerAlias sakai.server.other.1
> ServerAlias sakai.server.other.2
>
> DocumentRoot /var/www/html
>
> SetEnvIf client-ip ^NS$ REQ_FROM_LB
> SetEnvIf REQ_FROM_LB 1 DO_NOT_LOG
> SetEnvIf User-Agent nagios DO_NOT_LOG
>
> LogFormat "%{client-ip}i %l %u %t \"%r\" %>s %b \"%{Referer}i\" \"%{User-agent}i\" %v %A %D" combined_plus
>
> CustomLog "|cat >> /var/log/httpd/sakai-combined_log" combined_plus env=!DO_NOT_LOG
> ErrorLog "|cat >> /var/log/httpd/sakai-error_log"
>
> AcceptPathInfo on
>
> RedirectMatch permanent ^/$ https://sakai.server/portal
>
> # pass all requests through to local Tomcat on 8009 via AJP
> RewriteEngine On
> RewriteCond %{REQUEST_URI} !^/$
> RewriteRule ^/(.*) ajp://localhost:8009/$1 [P,L]
>
> </VirtualHost>
>
> Any help much appreciated.
>
> thanks,
> Steve
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://collab.sakaiproject.org/pipermail/production/attachments/20110728/2c2aacef/attachment-0001.html
More information about the production
mailing list