[Deploying Sakai] Urgent: Apache - Tomcat AJP issues

Steve Swinsburg steve.swinsburg at gmail.com
Wed Jul 27 23:06:49 PDT 2011 

Hi all,

I'm assisting diagnose a problem with a production Sakai instance that is experiencing a lot of outages since upgrading to 2.8.0 from 2.5.0. Almost all of the time it seems like something is happening at the Apache-Tomcat AJP connection which locks things up and requires frequent reboots (already three today).

The stacktraces all come from org.apache.jk

2011-07-28 14:15:43,824  WARN TP-Processor8 org.apache.jk.common.ChannelSocket - processCallbacks status 2
2011-07-28 14:15:43,836  WARN TP-Processor23 org.apache.jk.common.ChannelSocket - processCallbacks status 2
2011-07-28 14:15:43,864  WARN TP-Processor2 org.sakaiproject.content.impl.DbContentService - IOException 
java.io.IOException
	at org.apache.jk.common.JkInputStream.receive(JkInputStream.java:205)
	at org.apache.jk.common.JkInputStream.refillReadBuffer(JkInputStream.java:265)
	at org.apache.jk.common.JkInputStream.doRead(JkInputStream.java:183)
	at org.apache.coyote.Request.doRead(Request.java:419)
	at org.apache.catalina.connector.InputBuffer.realReadBytes(InputBuffer.java:270)
	at org.apache.tomcat.util.buf.ByteChunk.substract(ByteChunk.java:403)
	at org.apache.catalina.connector.InputBuffer.read(InputBuffer.java:293)
	at org.apache.catalina.connector.CoyoteInputStream.read(CoyoteInputStream.java:162)
	at org.sakaiproject.content.impl.DbContentService$DbStorage.putResourceBodyDb(DbContentService.java:2189)
	at org.sakaiproject.content.impl.DbContentService$DbStorage.commitResource(DbContentService.java:1745)
	at org.sakaiproject.content.impl.BaseContentService.commitResourceEdit(BaseContentService.java:5726)
	at org.sakaiproject.content.impl.BaseContentService.commitResource(BaseContentService.java:5539)
	at org.sakaiproject.dav.DavServlet.doPut(DavServlet.java:2779)
	at org.sakaiproject.dav.DavServlet.doDispatch(DavServlet.java:1139)
	at org.sakaiproject.dav.DavServlet.service(DavServlet.java:1066)
	at javax.servlet.http.HttpServlet.service(HttpServlet.java:729)
	at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:269)
	at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:188)
	at org.sakaiproject.util.RequestFilter.doFilter(RequestFilter.java:659)
	at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:215)
	at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:188)
	at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:213)
	at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:172)
	at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:563)
	at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:127)
	at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:117)
	at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:108)
	at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:174)
	at org.apache.jk.server.JkCoyoteHandler.invoke(JkCoyoteHandler.java:200)
	at org.apache.jk.common.HandlerRequest.invoke(HandlerRequest.java:291)
	at org.apache.jk.common.ChannelSocket.invoke(ChannelSocket.java:775)
	at org.apache.jk.common.ChannelSocket.processConnection(ChannelSocket.java:704)
	at org.apache.jk.common.ChannelSocket$SocketConnection.runIt(ChannelSocket.java:897)
	at org.apache.tomcat.util.threads.ThreadPool$ControlRunnable.run(ThreadPool.java:689)
	at java.lang.Thread.run(Thread.java:662)
2011-07-28 14:15:43,929  WARN TP-Processor14 org.apache.jk.common.ChannelSocket - processCallbacks status 2
2011-07-28 14:15:44,119  INFO main org.apache.catalina.core.StandardWrapper - Waiting for 3 instance(s) to be deallocated

2011-07-28 14:15:44,495  WARN TP-Processor12 org.apache.jk.common.ChannelSocket - processCallbacks status 2
2011-07-28 14:15:44,496  WARN TP-Processor18 org.apache.jk.core.MsgContext - Unable to send headers
java.net.SocketException: Broken pipe
	at java.net.SocketOutputStream.socketWrite0(Native Method)
	at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:92)
	at java.net.SocketOutputStream.write(SocketOutputStream.java:136)
	at org.apache.jk.common.ChannelSocket.send(ChannelSocket.java:538)
	at org.apache.jk.common.JkInputStream.appendHead(JkInputStream.java:326)
	at org.apache.jk.core.MsgContext.action(MsgContext.java:267)
	at org.apache.coyote.Response.action(Response.java:183)
	at org.apache.coyote.Response.sendHeaders(Response.java:379)
	at org.apache.catalina.connector.OutputBuffer.doFlush(OutputBuffer.java:313)
	at org.apache.catalina.connector.OutputBuffer.close(OutputBuffer.java:280)
	at org.apache.catalina.connector.Response.finishResponse(Response.java:484)
	at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:177)
	at org.apache.jk.server.JkCoyoteHandler.invoke(JkCoyoteHandler.java:200)
	at org.apache.jk.common.HandlerRequest.invoke(HandlerRequest.java:291)
	at org.apache.jk.common.ChannelSocket.invoke(ChannelSocket.java:775)
	at org.apache.jk.common.ChannelSocket.processConnection(ChannelSocket.java:704)
	at org.apache.jk.common.ChannelSocket$SocketConnection.runIt(ChannelSocket.java:897)
	at org.apache.tomcat.util.threads.ThreadPool$ControlRunnable.run(ThreadPool.java:689)
	at java.lang.Thread.run(Thread.java:662)


This time it was uploading a number of files in a folder (although total size only 4Mb).

The client has a standard Tomcat 8009 connector and then mod_proxy_ajp as per:

<VirtualHost *:80>
   ServerName  sakai.app.server
   ServerAlias sakai.server
   ServerAlias sakai.server.other.1
   ServerAlias sakai.server.other.2

   DocumentRoot        /var/www/html

   SetEnvIf client-ip ^NS$ REQ_FROM_LB
   SetEnvIf REQ_FROM_LB 1 DO_NOT_LOG
   SetEnvIf User-Agent nagios DO_NOT_LOG

   LogFormat "%{client-ip}i %l %u %t \"%r\" %>s %b \"%{Referer}i\" \"%{User-agent}i\" %v %A %D" combined_plus

   CustomLog "|cat >> /var/log/httpd/sakai-combined_log" combined_plus env=!DO_NOT_LOG
   ErrorLog "|cat >> /var/log/httpd/sakai-error_log"

   AcceptPathInfo on

   RedirectMatch permanent ^/$ https://sakai.server/portal

   # pass all requests through to local Tomcat on 8009 via AJP
   RewriteEngine   On
   RewriteCond     %{REQUEST_URI}  !^/$
   RewriteRule     ^/(.*)          ajp://localhost:8009/$1 [P,L]

</VirtualHost>

Any help much appreciated.

thanks,
Steve


-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://collab.sakaiproject.org/pipermail/production/attachments/20110728/089a9ace/attachment.html

More information about the production mailing list