[Deploying Sakai] [Building Sakai] 2.6 Production Issues

[Zhen Qian]

Michelle:

We are also aware of this during our load testing for 2.6 CTools  
release. As reported by Drew, "I think we identified that also in our  
load tests. The performance of this query seem to be not a problem  
but it's executed so many times." We use the default cache.

We will revisit and improve the caching for SAKAI_SITE_PAGE_PROPERTY,  
and hopefully have a patch for next CTools release.

Thanks,

- Zhen
On Aug 31, 2009, at 10:03 PM, Wagner, Michelle R. wrote:

> Hi all,
> Today IU experienced our heaviest load since upgrading to 2.6.   
> Starting last week (several of our campuses started last  
> Wednesday), Oncourse (our local Sakai brand) began experiencing  
> sporadic behavior where all of the application configured DB  
> connections were being used resulting in "An attempt by a client to  
> checkout a Connection has timed out" errors.  These errors  
> typically lasted about 30 seconds or so and then the pool recovered.
>
> Today all connections on all 8 of the servers were consumed.  In  
> response, we increased the maxPoolSize and restarted the servers.   
> This brought the application back, but we are still experiencing  
> short but high increases in the number of connections used by our  
> app servers.  At our peak usage, we had just over 200,000 requests  
> per 5 minutes.
>
> We are hoping to get some insight and/or recommendations from  
> others in the community, especially those of you who are running 2.6.
>
> * Is anyone else experiencing these short but high increases in  
> connection usage?
>
> * What connection pool settings are you using?  We currently have  
> the following.  We increased our maxPoolSize from 150 to 250 this  
> afternoon when our application went down:
> initialPoolSize at javax.sql.DataSource=50
> maxPoolSize at javax.sql.DataSource=250
> minPoolSize at javax.sql.DataSource=50
> maxConnectionAge at javax.sql.DataSource=0
> maxIdleTime at javax.sql.DataSource=0
> checkoutTimeout at javax.sql.DataSource=5000
> acquireRetryAttempts at javax.sql.DataSource=1
> maxIdleTimeExcessConnections at javax.sql.DataSource=300
> maxStatementsPerConnection at javax.sql.DataSource=0
> maxStatements at javax.sql.DataSource=0
>
> * We are still concerned about the query to  
> SAKAI_SITE_PAGE_PROPERTY.  It is being executed about 4x as many  
> times as any other query.  Does anyone have special cache settings  
> for this table?  What kind of results do you see in production?   
> Our table has no records in it.  We are using the default cache:
> [ name =  
> org.sakaiproject.db.BaseDbFlatStorage.SAKAI_SITE_PAGE_PROPERTY  
> status = STATUS_ALIVE eternal = false overflowToDisk = true  
> maxElementsInMemory = 10000 maxElementsOnDisk = 0  
> memoryStoreEvictionPolicy = LRU timeToLiveSeconds = 120  
> timeToIdleSeconds = 120 diskPersistent = false  
> diskExpiryThreadIntervalSeconds = 120 cacheEventListeners: hitCount  
> = 25624 memoryStoreHitCount = 25456 diskStoreHitCount = 168  
> missCountNotFound = 1743800 missCountExpired = 1680 ]
>
> * What are your top queries in terms of executions and/or cpu  
> time?  We want to get a better idea of what is "normal" for 2.6.
>
> Thanks so much for any help/insight you can provide,
> Michelle
> _______________________________________________
> sakai-dev mailing list
> sakai-dev at collab.sakaiproject.org
> http://collab.sakaiproject.org/mailman/listinfo/sakai-dev
>
> TO UNSUBSCRIBE: send email to sakai-dev- 
> unsubscribe at collab.sakaiproject.org with a subject of "unsubscribe"
>
>