[Building Sakai] Urgent: Apache - Tomcat AJP issues
Coetzee, Nico
Coetzeen at unisa.ac.za
Thu Jul 28 00:31:59 PDT 2011
This message (and attachments) is subject to restrictions and a disclaimer. Please refer to http://www.unisa.ac.za/disclaimer for full details.
________________________________
Hi Steve
We also had almost the same problem on our side. The tomcat's was responding fine, but Apache was dead. I had to decrease our tomcat's memory and made sure that the garbage collection is working. This could be the same problem you are experiencing
Cheers
On 28 Jul 2011, at 8:18 AM, Steve Swinsburg wrote:
Also, around the time of the latest outage I see a lot of these error messages in the Apache error_log
[Thu Jul 28 14:15:16 2011] [error] (70007)The timeout specified has expired: ajp_ilink_receive() can't receive header
[Thu Jul 28 14:15:20 2011] [error] (70007)The timeout specified has expired: ajp_ilink_receive() can't receive header
[Thu Jul 28 14:15:54 2011] [error] (104)Connection reset by peer: ajp_ilink_receive() can't receive header
Thanks,
Steve
On 28/07/2011, at 4:06 PM, Steve Swinsburg wrote:
Hi all,
I'm assisting diagnose a problem with a production Sakai instance that is experiencing a lot of outages since upgrading to 2.8.0 from 2.5.0. Almost all of the time it seems like something is happening at the Apache-Tomcat AJP connection which locks things up and requires frequent reboots (already three today).
The stacktraces all come from org.apache.jk
2011-07-28 14:15:43,824 WARN TP-Processor8 org.apache.jk.common.ChannelSocket - processCallbacks status 2
2011-07-28 14:15:43,836 WARN TP-Processor23 org.apache.jk.common.ChannelSocket - processCallbacks status 2
2011-07-28 14:15:43,864 WARN TP-Processor2 org.sakaiproject.content.impl.DbContentService - IOException
java.io.IOException
at org.apache.jk.common.JkInputStream.receive(JkInputStream.java:205)
at org.apache.jk.common.JkInputStream.refillReadBuffer(JkInputStream.java:265)
at org.apache.jk.common.JkInputStream.doRead(JkInputStream.java:183)
at org.apache.coyote.Request.doRead(Request.java:419)
at org.apache.catalina.connector.InputBuffer.realReadBytes(InputBuffer.java:270)
at org.apache.tomcat.util.buf.ByteChunk.substract(ByteChunk.java:403)
at org.apache.catalina.connector.InputBuffer.read(InputBuffer.java:293)
at org.apache.catalina.connector.CoyoteInputStream.read(CoyoteInputStream.java:162)
at org.sakaiproject.content.impl.DbContentService$DbStorage.putResourceBodyDb(DbContentService.java:2189)
at org.sakaiproject.content.impl.DbContentService$DbStorage.commitResource(DbContentService.java:1745)
at org.sakaiproject.content.impl.BaseContentService.commitResourceEdit(BaseContentService.java:5726)
at org.sakaiproject.content.impl.BaseContentService.commitResource(BaseContentService.java:5539)
at org.sakaiproject.dav.DavServlet.doPut(DavServlet.java:2779)
at org.sakaiproject.dav.DavServlet.doDispatch(DavServlet.java:1139)
at org.sakaiproject.dav.DavServlet.service(DavServlet.java:1066)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:729)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:269)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:188)
at org.sakaiproject.util.RequestFilter.doFilter(RequestFilter.java:659)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:215)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:188)
at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:213)
at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:172)
at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:563)
at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:127)
at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:117)
at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:108)
at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:174)
at org.apache.jk.server.JkCoyoteHandler.invoke(JkCoyoteHandler.java:200)
at org.apache.jk.common.HandlerRequest.invoke(HandlerRequest.java:291)
at org.apache.jk.common.ChannelSocket.invoke(ChannelSocket.java:775)
at org.apache.jk.common.ChannelSocket.processConnection(ChannelSocket.java:704)
at org.apache.jk.common.ChannelSocket$SocketConnection.runIt(ChannelSocket.java:897)
at org.apache.tomcat.util.threads.ThreadPool$ControlRunnable.run(ThreadPool.java:689)
at java.lang.Thread.run(Thread.java:662)
2011-07-28 14:15:43,929 WARN TP-Processor14 org.apache.jk.common.ChannelSocket - processCallbacks status 2
2011-07-28 14:15:44,119 INFO main org.apache.catalina.core.StandardWrapper - Waiting for 3 instance(s) to be deallocated
2011-07-28 14:15:44,495 WARN TP-Processor12 org.apache.jk.common.ChannelSocket - processCallbacks status 2
2011-07-28 14:15:44,496 WARN TP-Processor18 org.apache.jk.core.MsgContext - Unable to send headers
java.net.SocketException: Broken pipe
at java.net.SocketOutputStream.socketWrite0(Native Method)
at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:92)
at java.net.SocketOutputStream.write(SocketOutputStream.java:136)
at org.apache.jk.common.ChannelSocket.send(ChannelSocket.java:538)
at org.apache.jk.common.JkInputStream.appendHead(JkInputStream.java:326)
at org.apache.jk.core.MsgContext.action(MsgContext.java:267)
at org.apache.coyote.Response.action(Response.java:183)
at org.apache.coyote.Response.sendHeaders(Response.java:379)
at org.apache.catalina.connector.OutputBuffer.doFlush(OutputBuffer.java:313)
at org.apache.catalina.connector.OutputBuffer.close(OutputBuffer.java:280)
at org.apache.catalina.connector.Response.finishResponse(Response.java:484)
at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:177)
at org.apache.jk.server.JkCoyoteHandler.invoke(JkCoyoteHandler.java:200)
at org.apache.jk.common.HandlerRequest.invoke(HandlerRequest.java:291)
at org.apache.jk.common.ChannelSocket.invoke(ChannelSocket.java:775)
at org.apache.jk.common.ChannelSocket.processConnection(ChannelSocket.java:704)
at org.apache.jk.common.ChannelSocket$SocketConnection.runIt(ChannelSocket.java:897)
at org.apache.tomcat.util.threads.ThreadPool$ControlRunnable.run(ThreadPool.java:689)
at java.lang.Thread.run(Thread.java:662)
This time it was uploading a number of files in a folder (although total size only 4Mb).
The client has a standard Tomcat 8009 connector and then mod_proxy_ajp as per:
<VirtualHost *:80>
ServerName sakai.app.server
ServerAlias sakai.server
ServerAlias sakai.server.other.1
ServerAlias sakai.server.other.2
DocumentRoot /var/www/html
SetEnvIf client-ip ^NS$ REQ_FROM_LB
SetEnvIf REQ_FROM_LB 1 DO_NOT_LOG
SetEnvIf User-Agent nagios DO_NOT_LOG
LogFormat "%{client-ip}i %l %u %t \"%r\" %>s %b \"%{Referer}i\" \"%{User-agent}i\" %v %A %D" combined_plus
CustomLog "|cat >> /var/log/httpd/sakai-combined_log" combined_plus env=!DO_NOT_LOG
ErrorLog "|cat >> /var/log/httpd/sakai-error_log"
AcceptPathInfo on
RedirectMatch permanent ^/$ https://sakai.server/portal
# pass all requests through to local Tomcat on 8009 via AJP
RewriteEngine On
RewriteCond %{REQUEST_URI} !^/$
RewriteRule ^/(.*) ajp://localhost:8009/$1 [P,L]
</VirtualHost>
Any help much appreciated.
thanks,
Steve
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://collab.sakaiproject.org/pipermail/sakai-dev/attachments/20110728/3fb02086/attachment.html
-------------- next part --------------
An embedded and charset-unspecified text was scrubbed...
Name: ATT00001..txt
Url: http://collab.sakaiproject.org/pipermail/sakai-dev/attachments/20110728/3fb02086/attachment.txt
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://collab.sakaiproject.org/pipermail/sakai-dev/attachments/20110728/3fb02086/attachment.htm
-------------- next part --------------
A non-text attachment was scrubbed...
Name: Unisa-email-signature_UNGC.gif
Type: image/gif
Size: 8678 bytes
Desc: Unisa-email-signature_UNGC.gif
Url : http://collab.sakaiproject.org/pipermail/sakai-dev/attachments/20110728/3fb02086/attachment.gif
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://collab.sakaiproject.org/pipermail/sakai-dev/attachments/20110728/3fb02086/attachment-0001.htm
-------------- next part --------------
A non-text attachment was scrubbed...
Name: image002.gif
Type: image/gif
Size: 4902 bytes
Desc: image002.gif
Url : http://collab.sakaiproject.org/pipermail/sakai-dev/attachments/20110728/3fb02086/attachment-0001.gif
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://collab.sakaiproject.org/pipermail/sakai-dev/attachments/20110728/3fb02086/attachment-0002.htm
More information about the sakai-dev
mailing list