[Building Sakai] Fwd: CAS authentication issue related with apache down

[Sanghyun Jeon]

Here's some more diggings.

When SSO has a large number of requests, Sakai starts to complain about CAS
authentication and then all sakai tomcat logs keep throwing

[DEBUG] org.sakaiproject.user.impl.AuthenticationCache:88 -
getAuthentication: returning record for authenticationId=xxxx

repeatedly at the same time.
Then it throws TicketValidationException until we restart apache server.

our cacti shows apache CPU is very low during this incident. When we
restart apache server, the abnormal spike of CPU usage is shown.
We are using Apache as front end with load balancer (prepork) and there are
3 Sakai app servers with 2 tomcat app nodes per each server behind that
apache server.
We followed CASifying Sakai instruction when we integrated with CAS.

I hope somebody can shed some light on why all Sakai tomcat nodes are
returning this specific user's record? Is this the last user to send an
authentication request to CAS?
Is this authentication caching issue?

Frankly speaking I cannot narrow down the issue so that I can not resolve
this issue yet,

S





---------- Forwarded message ----------
From: Sanghyun Jeon <euksa99 at gmail.com>
Date: Mon, Dec 15, 2014 at 11:57 AM
Subject: CAS authentication issue related with apache down
To: sakai-dev <sakai-dev at collab.sakaiproject.org>

Hello All,

Last night our Sakai apache did not respond due to socket timeout.

While we investigated the incident, our tomcat log showed we had CAS
authentication issue. Our CAS team confirmed that CAS had a high volume of
requests at the same timeframe.
our Apache error log showed "server reached MaxClients setting, consider
raising the MaxClients setting."

My question is that can we gracefully handle CAS authentication failure
with appropriate MaxClient setting?

Are they related each other?


Any advice /suggestion would be greatly appreciated.
S
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://collab.sakaiproject.org/pipermail/sakai-dev/attachments/20141215/d8bec536/attachment.html