[Building Sakai] Strange request pattern incidents in production

[Jon Higham]

Thanks for getting back to me.

 

Sorry, it should be mneme 2.1.2 with utils 1.0.8 and ambrosia 2.1.2. I'm
having similar problems with melete 2.8.2 as well and got my versions
mixed. 

 

I have etudes-ambrosia-api-2.1.2.jar in shared/lib. I've even tried
copying from a working version of sakai 2.8.

 

I also get a similar error for org.etudes.mneme.api.MnemeService, but
again I have etudes-mneme-api-2.1.2.jar in shared/lib.

 

I can't see any other instances of either etudes-ambrosia-api anywhere
in tomcat 7.0.21.

 

 

 

Jon

 

 

From: sakai-dev-bounces at collab.sakaiproject.org
[mailto:sakai-dev-bounces at collab.sakaiproject.org] On Behalf Of Glenn R.
Golden
Sent: 15 December 2011 19:01
To: Sakai Dev
Subject: [Building Sakai] Strange request pattern incidents in
production

 

We have had two production "incidents" recently.  The symptoms are an
app server being slow, or the database heating up, or an app server
using an unusually large number of open files (which we monitor), or a
raft of bug report emails.  The problem persists for a few or 10s of
minutes, and is limited to only one of the app servers in the cluster
(one at a time).

 

The real problem, though, after looking at the access logs for the
period, is a huge series of duplicate requests from a single user over a
short period of time.  In the first incident, we got 3000 requests over
20 seconds.  In the second one, we got 1400 over 4 minutes.  In each
series, they are all the same request, to the same app server, from the
same user.  The requests look normal - one was a Melete list request and
one was a Mneme list request (these are two of our most used tools).  It
is the number of them that is the problem.

 

Of course, Sakai will process all of these requests, even though the
browser has abandoned the connection.  The first one processes in under
a second.  By the time the last one is done, it has taken 100s of
seconds to process.

 

This one user's barrage of requests slows down the app server, puts an
increased load on the database, and keeps an abnormally large number of
concurrent requests in Apache and Tomcat.  This increase in the number
of concurrent requests is what we see in the increased file usage, as
the Apache - Tomcat connector uses a unix "file" for each connection.
In once case, it stressed the db connection pool on the app server to
the point that it stopped delivering connections (or the clients all
timed out waiting).

 

Once the duplicate request series is done, and the requests are
processed, the app server quickly goes back to normal.

 

I'm wondering: has anyone else seen anything like this?

 

The browser in both of these cases was Chrome on Windows.

 

- Glenn

 

Glenn R. Golden

Chief Architect, Etudes, Inc.

ggolden at etudes.org

 

-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://collab.sakaiproject.org/pipermail/sakai-dev/attachments/20120323/820af68a/attachment.html 
-------------- next part --------------
**************************************************
To view the terms under which this email is 
distributed, please go to 
http://www2.hull.ac.uk/legal/disclaimer.aspx
**************************************************