[Deploying Sakai] [Building Sakai] 2.6 Production Issues

[Speelmon, Lance Day]

Michelle,

Did you implement the change I suggested yesterday morning to disable  
all database reads from SAKAI_SITE_PAGE_PROPERTY to alleviate the load  
and give you some breathing room?  Since you have zero rows in this  
table, that seemed like a very safe thing to do (i.e. dealing with the  
immediate).  If so, can you share the patch you applied?

Next, I spent about four hours last night trying to search, grep, and  
review svn logs to determine what calls to SitePage.getProperties have  
changed since 2.5.  So far, I have nothing conclusive to share as this  
methods extends from Entity.getProperties and Eclipse's get references/ 
call hierarchy is finding way too many calls to Entity.getProperties  
to be of any use.  To expedite things, I would make the following  
change to track down all of the places that are now calling  
SitePage.getProperties under normal use.  Obviously you will only want  
to run this in a test environment:

Index: kernel-impl/src/main/java/org/sakaiproject/site/impl/ 
DbSiteService.java
===================================================================
--- kernel-impl/src/main/java/org/sakaiproject/site/impl/ 
DbSiteService.java	(revision 66237)
+++ kernel-impl/src/main/java/org/sakaiproject/site/impl/ 
DbSiteService.java	(working copy)
@@ -1860,6 +1860,7 @@
  		 */
  		public void readPageProperties(SitePage page,  
ResourcePropertiesEdit props)
  		{
+			System.out.println(new Error("SAKAI_SITE_PAGE_PROPERTY trace"));
  			boolean makeCall = ServerConfigurationService.getBoolean 
("query.run.SAKAI_SITE_PAGE_PROPERTY", true);
  			if (makeCall) {
  				super.readProperties(null, "SAKAI_SITE_PAGE_PROPERTY",  
"PAGE_ID", page.getId(), props);


Run through your normal uses and examine the stack traces.  This will  
at least tell you if the changes since 2.5 are related to a) the i18n  
of page titles, b) IU's new Messages and Forums synoptic tool, c)  
something else we have not thought of yet, or d) some combination of  
the above.  Then code optimization/profiling can then occur on those  
code paths.  L

Lance Speelmon
Scholarly Technologist

On Aug 31, 2009, at 10:03 PM, Wagner, Michelle R. wrote:

> Hi all,
> Today IU experienced our heaviest load since upgrading to 2.6.   
> Starting last week (several of our campuses started last Wednesday),  
> Oncourse (our local Sakai brand) began experiencing sporadic  
> behavior where all of the application configured DB connections were  
> being used resulting in "An attempt by a client to checkout a  
> Connection has timed out" errors.  These errors typically lasted  
> about 30 seconds or so and then the pool recovered.
>
> Today all connections on all 8 of the servers were consumed.  In  
> response, we increased the maxPoolSize and restarted the servers.   
> This brought the application back, but we are still experiencing  
> short but high increases in the number of connections used by our  
> app servers.  At our peak usage, we had just over 200,000 requests  
> per 5 minutes.
>
> We are hoping to get some insight and/or recommendations from others  
> in the community, especially those of you who are running 2.6.
>
> * Is anyone else experiencing these short but high increases in  
> connection usage?
>
> * What connection pool settings are you using?  We currently have  
> the following.  We increased our maxPoolSize from 150 to 250 this  
> afternoon when our application went down:
> initialPoolSize at javax.sql.DataSource=50
> maxPoolSize at javax.sql.DataSource=250
> minPoolSize at javax.sql.DataSource=50
> maxConnectionAge at javax.sql.DataSource=0
> maxIdleTime at javax.sql.DataSource=0
> checkoutTimeout at javax.sql.DataSource=5000
> acquireRetryAttempts at javax.sql.DataSource=1
> maxIdleTimeExcessConnections at javax.sql.DataSource=300
> maxStatementsPerConnection at javax.sql.DataSource=0
> maxStatements at javax.sql.DataSource=0
>
> * We are still concerned about the query to  
> SAKAI_SITE_PAGE_PROPERTY.  It is being executed about 4x as many  
> times as any other query.  Does anyone have special cache settings  
> for this table?  What kind of results do you see in production?  Our  
> table has no records in it.  We are using the default cache:
> [ name =  
> org.sakaiproject.db.BaseDbFlatStorage.SAKAI_SITE_PAGE_PROPERTY  
> status = STATUS_ALIVE eternal = false overflowToDisk = true  
> maxElementsInMemory = 10000 maxElementsOnDisk = 0  
> memoryStoreEvictionPolicy = LRU timeToLiveSeconds = 120  
> timeToIdleSeconds = 120 diskPersistent = false  
> diskExpiryThreadIntervalSeconds = 120 cacheEventListeners: hitCount  
> = 25624 memoryStoreHitCount = 25456 diskStoreHitCount = 168  
> missCountNotFound = 1743800 missCountExpired = 1680 ]
>
> * What are your top queries in terms of executions and/or cpu time?   
> We want to get a better idea of what is "normal" for 2.6.
>
> Thanks so much for any help/insight you can provide,
> Michelle
> _______________________________________________
> sakai-dev mailing list
> sakai-dev at collab.sakaiproject.org
> http://collab.sakaiproject.org/mailman/listinfo/sakai-dev
>
> TO UNSUBSCRIBE: send email to sakai-dev-unsubscribe at collab.sakaiproject.org 
>  with a subject of "unsubscribe"