[Deploying Sakai] Urgent: Apache - Tomcat AJP issues

[Matthew Buckett]

On Thu, Jul 28, 2011 at 7:06 AM, Steve Swinsburg
<steve.swinsburg at gmail.com> wrote:
> Hi all,
> I'm assisting diagnose a problem with a production Sakai instance that is
> experiencing a lot of outages since upgrading to 2.8.0 from 2.5.0. Almost
> all of the time it seems like something is happening at the Apache-Tomcat
> AJP connection which locks things up and requires frequent reboots (already
> three today).

Did you change the version of Apache or Tomcat during your upgrade
from 2.5.0 to 2.8.0? We've has issues with old versions of Apache with
mod_proxy_ajp locking up under high load.

We could manage to reproduce this by making 100s of requests a second
and a Tomcat instance ran out of workers it's AJP link to Apache would
lockup and no more requests would get handled by that node, they would
just timeout. Our first fix was to reduce the keepalive time so that
we weren't holding as many Tomcat workers open.

Secondly we are going to move to a newer version of Apache which seems
to have fixed the underlying issue.

Do you have mod_status installed? We were seeing lots of scoreboard
entries in the Sending Reply state (W) when this was happening. We
also had this in our logs (only once though):

http://article.gmane.org/gmane.comp.cms.sakai.devel/17927

-- 
  Matthew Buckett
  VLE Developer, LTG, Oxford University Computing Services