[Building Sakai] Strange request pattern incidents in production

[Glenn R. Golden]

We have had two production "incidents" recently.  The symptoms are an app server being slow, or the database heating up, or an app server using an unusually large number of open files (which we monitor), or a raft of bug report emails.  The problem persists for a few or 10s of minutes, and is limited to only one of the app servers in the cluster (one at a time).

The real problem, though, after looking at the access logs for the period, is a huge series of duplicate requests from a single user over a short period of time.  In the first incident, we got 3000 requests over 20 seconds.  In the second one, we got 1400 over 4 minutes.  In each series, they are all the same request, to the same app server, from the same user.  The requests look normal - one was a Melete list request and one was a Mneme list request (these are two of our most used tools).  It is the number of them that is the problem.

Of course, Sakai will process all of these requests, even though the browser has abandoned the connection.  The first one processes in under a second.  By the time the last one is done, it has taken 100s of seconds to process.

This one user's barrage of requests slows down the app server, puts an increased load on the database, and keeps an abnormally large number of concurrent requests in Apache and Tomcat.  This increase in the number of concurrent requests is what we see in the increased file usage, as the Apache - Tomcat connector uses a unix "file" for each connection.  In once case, it stressed the db connection pool on the app server to the point that it stopped delivering connections (or the clients all timed out waiting).

Once the duplicate request series is done, and the requests are processed, the app server quickly goes back to normal.

I'm wondering: has anyone else seen anything like this?

The browser in both of these cases was Chrome on Windows.

- Glenn

Glenn R. Golden
Chief Architect, Etudes, Inc.
ggolden at etudes.org

-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://collab.sakaiproject.org/pipermail/sakai-dev/attachments/20111215/bb217079/attachment.html