[Deploying Sakai] FW: 2.6 Production Issues

[May, Megan Marie]

Cross posting for Michelle since she's not on this list (yet!) 

-----Original Message-----
From: sakai-dev-bounces at collab.sakaiproject.org [mailto:sakai-dev-bounces at collab.sakaiproject.org] On Behalf Of Wagner, Michelle R.
Sent: Monday, August 31, 2009 10:04 PM
To: sakai-dev at collab.sakaiproject.org; production at collab.sakaiproject.org
Subject: [Building Sakai] 2.6 Production Issues

Hi all,
Today IU experienced our heaviest load since upgrading to 2.6.  Starting last week (several of our campuses started last Wednesday), Oncourse (our local Sakai brand) began experiencing sporadic behavior where all of the application configured DB connections were being used resulting in "An attempt by a client to checkout a Connection has timed out" errors.  These errors typically lasted about 30 seconds or so and then the pool recovered. 

Today all connections on all 8 of the servers were consumed.  In response, we increased the maxPoolSize and restarted the servers.  This brought the application back, but we are still experiencing short but high increases in the number of connections used by our app servers.  At our peak usage, we had just over 200,000 requests per 5 minutes.

We are hoping to get some insight and/or recommendations from others in the community, especially those of you who are running 2.6.

* Is anyone else experiencing these short but high increases in connection usage?

* What connection pool settings are you using?  We currently have the following.  We increased our maxPoolSize from 150 to 250 this afternoon when our application went down:
initialPoolSize at javax.sql.DataSource=50
maxPoolSize at javax.sql.DataSource=250
minPoolSize at javax.sql.DataSource=50
maxConnectionAge at javax.sql.DataSource=0
maxIdleTime at javax.sql.DataSource=0
checkoutTimeout at javax.sql.DataSource=5000
acquireRetryAttempts at javax.sql.DataSource=1
maxIdleTimeExcessConnections at javax.sql.DataSource=300
maxStatementsPerConnection at javax.sql.DataSource=0
maxStatements at javax.sql.DataSource=0

* We are still concerned about the query to SAKAI_SITE_PAGE_PROPERTY.  It is being executed about 4x as many times as any other query.  Does anyone have special cache settings for this table?  What kind of results do you see in production?  Our table has no records in it.  We are using the default cache:
[ name = org.sakaiproject.db.BaseDbFlatStorage.SAKAI_SITE_PAGE_PROPERTY status = STATUS_ALIVE eternal = false overflowToDisk = true maxElementsInMemory = 10000 maxElementsOnDisk = 0 memoryStoreEvictionPolicy = LRU timeToLiveSeconds = 120 timeToIdleSeconds = 120 diskPersistent = false diskExpiryThreadIntervalSeconds = 120 cacheEventListeners: hitCount = 25624 memoryStoreHitCount = 25456 diskStoreHitCount = 168 missCountNotFound = 1743800 missCountExpired = 1680 ]

* What are your top queries in terms of executions and/or cpu time?  We want to get a better idea of what is "normal" for 2.6.

Thanks so much for any help/insight you can provide, 
Michelle
_______________________________________________
sakai-dev mailing list
sakai-dev at collab.sakaiproject.org
http://collab.sakaiproject.org/mailman/listinfo/sakai-dev

TO UNSUBSCRIBE: send email to sakai-dev-unsubscribe at collab.sakaiproject.org with a subject of "unsubscribe"