[Building Sakai] Strange request pattern incidents in production

[Mark Norton]

Sure sounds like a DOS attack to me.

- Mark

On 12/15/2011 2:01 PM, Glenn R. Golden wrote:
> We have had two production "incidents" recently.  The symptoms are an 
> app server being slow, or the database heating up, or an app server 
> using an unusually large number of open files (which we monitor), or a 
> raft of bug report emails.  The problem persists for a few or 10s of 
> minutes, and is limited to only one of the app servers in the cluster 
> (one at a time).
>
> The real problem, though, after looking at the access logs for the 
> period, is a huge series of duplicate requests from a single user over 
> a short period of time.  In the first incident, we got 3000 requests 
> over 20 seconds.  In the second one, we got 1400 over 4 minutes.  In 
> each series, they are all the same request, to the same app server, 
> from the same user.  The requests look normal - one was a Melete list 
> request and one was a Mneme list request (these are two of our most 
> used tools).  It is the number of them that is the problem.
>
> Of course, Sakai will process all of these requests, even though the 
> browser has abandoned the connection.  The first one processes in 
> under a second.  By the time the last one is done, it has taken 100s 
> of seconds to process.
>
> This one user's barrage of requests slows down the app server, puts an 
> increased load on the database, and keeps an abnormally large number 
> of concurrent requests in Apache and Tomcat.  This increase in the 
> number of concurrent requests is what we see in the increased file 
> usage, as the Apache - Tomcat connector uses a unix "file" for each 
> connection.  In once case, it stressed the db connection pool on the 
> app server to the point that it stopped delivering connections (or the 
> clients all timed out waiting).
>
> Once the duplicate request series is done, and the requests are 
> processed, the app server quickly goes back to normal.
>
> I'm wondering: has anyone else seen anything like this?
>
> The browser in both of these cases was Chrome on Windows.
>
> - Glenn
>
> Glenn R. Golden
> Chief Architect, Etudes, Inc.
> ggolden at etudes.org <mailto:ggolden at etudes.org>
>
>
>
> _______________________________________________
> sakai-dev mailing list
> sakai-dev at collab.sakaiproject.org
> http://collab.sakaiproject.org/mailman/listinfo/sakai-dev
>
> TO UNSUBSCRIBE: send email to sakai-dev-unsubscribe at collab.sakaiproject.org with a subject of "unsubscribe"

-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://collab.sakaiproject.org/pipermail/sakai-dev/attachments/20111215/5a309279/attachment.html