[Deploying Sakai] [Building Sakai] Urgent: Apache - Tomcat AJP issues

Robert Cohen robert.cohen at anu.edu.au
Thu Jul 28 01:38:45 PDT 2011


That¹s seems like an odd way of proxing to ajp.
Is that even using mod_proxy_ajp?

The way we're doing It is

ProxyPassMatch ^/$ !
ProxyPassMatch ^/library !
ProxyPassMatch ^/help !
ProxyPassMatch ^/welcome !
ProxyPassMatch ^/index.html$ !
ProxyPass / ajp://localhost:8009/
ProxyPassReverse / ajp://localhost:8009/


=======================================
Robert Cohen
Information Technology Infrastructure
Division of Information
R.G Menzies Building
Building 2
The Australian National University
Canberra ACT 0200 Australia
 
T: +61 2 6125 8389
F: +61 2 6125 7699
http://www.anu.edu.au
 
CRICOS Provider #00120C
=======================================

From:  Steve Swinsburg <steve.swinsburg at gmail.com>
Date:  Thu, 28 Jul 2011 16:06:49 +1000
To:  Sakai Dev <sakai-dev at collab.sakaiproject.org>, production List
<production at collab.sakaiproject.org>
Subject:  [Building Sakai] Urgent: Apache - Tomcat AJP issues

> Hi all,
> 
> I'm assisting diagnose a problem with a production Sakai instance that is
> experiencing a lot of outages since upgrading to 2.8.0 from 2.5.0. Almost all
> of the time it seems like something is happening at the Apache-Tomcat AJP
> connection which locks things up and requires frequent reboots (already three
> today).
> 
> The stacktraces all come from org.apache.jk
> 
> 2011-07-28 14:15:43,824  WARN TP-Processor8 org.apache.jk.common.ChannelSocket
> - processCallbacks status 2
> 2011-07-28 14:15:43,836  WARN TP-Processor23
> org.apache.jk.common.ChannelSocket - processCallbacks status 2
> 2011-07-28 14:15:43,864  WARN TP-Processor2
> org.sakaiproject.content.impl.DbContentService - IOException
> java.io.IOException
> at org.apache.jk.common.JkInputStream.receive(JkInputStream.java:205)
> at org.apache.jk.common.JkInputStream.refillReadBuffer(JkInputStream.java:265)
> at org.apache.jk.common.JkInputStream.doRead(JkInputStream.java:183)
> at org.apache.coyote.Request.doRead(Request.java:419)
> at 
> org.apache.catalina.connector.InputBuffer.realReadBytes(InputBuffer.java:270)
> at org.apache.tomcat.util.buf.ByteChunk.substract(ByteChunk.java:403)
> at org.apache.catalina.connector.InputBuffer.read(InputBuffer.java:293)
> at 
> org.apache.catalina.connector.CoyoteInputStream.read(CoyoteInputStream.java:16
> 2)
> at 
> org.sakaiproject.content.impl.DbContentService$DbStorage.putResourceBodyDb(DbC
> ontentService.java:2189)
> at 
> org.sakaiproject.content.impl.DbContentService$DbStorage.commitResource(DbCont
> entService.java:1745)
> at 
> org.sakaiproject.content.impl.BaseContentService.commitResourceEdit(BaseConten
> tService.java:5726)
> at 
> org.sakaiproject.content.impl.BaseContentService.commitResource(BaseContentSer
> vice.java:5539)
> at org.sakaiproject.dav.DavServlet.doPut(DavServlet.java:2779)
> at org.sakaiproject.dav.DavServlet.doDispatch(DavServlet.java:1139)
> at org.sakaiproject.dav.DavServlet.service(DavServlet.java:1066)
> at javax.servlet.http.HttpServlet.service(HttpServlet.java:729)
> at 
> org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFi
> lterChain.java:269)
> at 
> org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChai
> n.java:188)
> at org.sakaiproject.util.RequestFilter.doFilter(RequestFilter.java:659)
> at 
> org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFi
> lterChain.java:215)
> at 
> org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChai
> n.java:188)
> at 
> org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java
> :213)
> at 
> org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java
> :172)
> at 
> org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.j
> ava:563)
> at 
> org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:127)
> at 
> org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:117)
> at 
> org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:1
> 08)
> at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:174)
> at org.apache.jk.server.JkCoyoteHandler.invoke(JkCoyoteHandler.java:200)
> at org.apache.jk.common.HandlerRequest.invoke(HandlerRequest.java:291)
> at org.apache.jk.common.ChannelSocket.invoke(ChannelSocket.java:775)
> at 
> org.apache.jk.common.ChannelSocket.processConnection(ChannelSocket.java:704)
> at 
> org.apache.jk.common.ChannelSocket$SocketConnection.runIt(ChannelSocket.java:8
> 97)
> at 
> org.apache.tomcat.util.threads.ThreadPool$ControlRunnable.run(ThreadPool.java:
> 689)
> at java.lang.Thread.run(Thread.java:662)
> 2011-07-28 14:15:43,929  WARN TP-Processor14
> org.apache.jk.common.ChannelSocket - processCallbacks status 2
> 2011-07-28 14:15:44,119  INFO main org.apache.catalina.core.StandardWrapper -
> Waiting for 3 instance(s) to be deallocated
> 
> 2011-07-28 14:15:44,495  WARN TP-Processor12
> org.apache.jk.common.ChannelSocket - processCallbacks status 2
> 2011-07-28 14:15:44,496  WARN TP-Processor18 org.apache.jk.core.MsgContext -
> Unable to send headers
> java.net.SocketException: Broken pipe
> at java.net.SocketOutputStream.socketWrite0(Native Method)
> at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:92)
> at java.net.SocketOutputStream.write(SocketOutputStream.java:136)
> at org.apache.jk.common.ChannelSocket.send(ChannelSocket.java:538)
> at org.apache.jk.common.JkInputStream.appendHead(JkInputStream.java:326)
> at org.apache.jk.core.MsgContext.action(MsgContext.java:267)
> at org.apache.coyote.Response.action(Response.java:183)
> at org.apache.coyote.Response.sendHeaders(Response.java:379)
> at org.apache.catalina.connector.OutputBuffer.doFlush(OutputBuffer.java:313)
> at org.apache.catalina.connector.OutputBuffer.close(OutputBuffer.java:280)
> at org.apache.catalina.connector.Response.finishResponse(Response.java:484)
> at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:177)
> at org.apache.jk.server.JkCoyoteHandler.invoke(JkCoyoteHandler.java:200)
> at org.apache.jk.common.HandlerRequest.invoke(HandlerRequest.java:291)
> at org.apache.jk.common.ChannelSocket.invoke(ChannelSocket.java:775)
> at 
> org.apache.jk.common.ChannelSocket.processConnection(ChannelSocket.java:704)
> at 
> org.apache.jk.common.ChannelSocket$SocketConnection.runIt(ChannelSocket.java:8
> 97)
> at 
> org.apache.tomcat.util.threads.ThreadPool$ControlRunnable.run(ThreadPool.java:
> 689)
> at java.lang.Thread.run(Thread.java:662)
> 
> 
> This time it was uploading a number of files in a folder (although total size
> only 4Mb).
> 
> The client has a standard Tomcat 8009 connector and then mod_proxy_ajp as per:
> 
> <VirtualHost *:80>
>    ServerName  sakai.app.server
>    ServerAlias sakai.server
>    ServerAlias sakai.server.other.1
>    ServerAlias sakai.server.other.2
> 
>    DocumentRoot        /var/www/html
> 
>    SetEnvIf client-ip ^NS$ REQ_FROM_LB
>    SetEnvIf REQ_FROM_LB 1 DO_NOT_LOG
>    SetEnvIf User-Agent nagios DO_NOT_LOG
> 
>    LogFormat "%{client-ip}i %l %u %t \"%r\" %>s %b \"%{Referer}i\"
> \"%{User-agent}i\" %v %A %D" combined_plus
> 
>    CustomLog "|cat >> /var/log/httpd/sakai-combined_log" combined_plus
> env=!DO_NOT_LOG
>    ErrorLog "|cat >> /var/log/httpd/sakai-error_log"
> 
>    AcceptPathInfo on
> 
>    RedirectMatch permanent ^/$ https://sakai.server/portal
> 
>    # pass all requests through to local Tomcat on 8009 via AJP
>    RewriteEngine   On
>    RewriteCond     %{REQUEST_URI}  !^/$
>    RewriteRule     ^/(.*)          ajp://localhost:8009/$1 [P,L]
> 
> </VirtualHost>
> 
> Any help much appreciated.
> 
> thanks,
> Steve
> 
> 
> _______________________________________________ sakai-dev mailing list
> sakai-dev at collab.sakaiproject.org
> http://collab.sakaiproject.org/mailman/listinfo/sakai-dev TO UNSUBSCRIBE: send
> email to sakai-dev-unsubscribe at collab.sakaiproject.org with a subject of
> "unsubscribe"


-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://collab.sakaiproject.org/pipermail/production/attachments/20110728/5194a6ed/attachment-0001.html 


More information about the production mailing list