[Building Sakai] Production Sakai 2.7.0 deadlocks

[David Barroso]

Hi all,

We have a Sakai 2.7.0 instance in production using
Oracle 10g, Java1.6 and Linux.

When a high level of sessions is reached, the
java garbage collector starts working and the
following error is shown in catalina.out

Once the GregorSamsa class is unloaded and the
java.lang.IndexOutOfBoundsException appears,
the servers enters in a deadlock, waiting for
database connections

org.apache.commons.dbcp.SQLNestedException: Cannot get a connection, 
pool error Timeout waiting for idle object
Caused by: java.util.NoSuchElementException: Timeout waiting for idle object

We have found Oracle and Tomcat bug reports
talking about this, but in very old versions.

Any idea or clue to solve this problem ?

I've been watching the nightly2 oracle instance and
the Unloading class GregorSamsa appears at the end
of the catalina.out log.

http://nightly2.sakaiproject.org/logs/tomcat-nightly-oracle/

Regards,


-------------
our catalina.out

[Unloading class GregorSamsa]
[Unloading class GregorSamsa$2]
[Unloading class GregorSamsa$0]
[Unloading class GregorSamsa]
[Unloading class GregorSamsa$4]
[Unloading class GregorSamsa$0]
[Unloading class GregorSamsa]
[Unloading class GregorSamsa]
[Unloading class GregorSamsa$2]
[Unloading class GregorSamsa$3]
[Unloading class GregorSamsa$4]
[Unloading class GregorSamsa$1]
2010-08-04 12:17:07,075  WARN TP-Processor17 
org.apache.myfaces.application.jsp.JspViewHandlerImpl - no faces servlet 
mappings found
2010-08-04 12:17:08,841  INFO TP-Processor10 
org.apache.myfaces.shared_impl.config.MyfacesConfig - Tomahawk jar not 
available. Autoscrolling, DetectJavascript, AddResourceClass and 
CheckExtensionsFilter are disabled now.
2010-08-04 12:17:08,841  INFO TP-Processor10 
org.apache.myfaces.shared_impl.config.MyfacesConfig - Starting up 
Tomahawk on the MyFaces-JSF-Implementation
2010-08-04 12:17:12,435 ERROR http-8080-Processor61 
org.apache.catalina.core.ContainerBase.[Catalina].[localhost].[/sakai-site-manage-tool].[sakai.membership] 
- Servlet.service() for servlet sakai.membership threw exception
java.lang.ArrayIndexOutOfBoundsException: -31
         at 
sun.util.calendar.BaseCalendar.getCalendarDateFromFixedDate(BaseCalendar.java:436)
         at 
java.util.GregorianCalendar.computeFields(GregorianCalendar.java:2081)
         at 
java.util.GregorianCalendar.computeTime(GregorianCalendar.java:2472)
         at java.util.Calendar.updateTime(Calendar.java:2468)
         at java.util.Calendar.getTimeInMillis(Calendar.java:1087)
         at 
oracle.jdbc.driver.DateTimeCommonAccessor.getTimestamp(DateTimeCommonAccessor.java:256)
         at 
oracle.jdbc.driver.OracleResultSetImpl.getTimestamp(OracleResultSetImpl.java:1644)
         at 
org.apache.commons.dbcp.DelegatingResultSet.getTimestamp(DelegatingResultSet.java:548)
         at 
org.sakaiproject.site.impl.DbSiteService$DbStorage.readSqlResultRecord(DbSiteService.java:2128)
         at 
org.sakaiproject.db.impl.BasicSqlService.dbRead(BasicSqlService.java:564)
         at 
org.sakaiproject.db.impl.BasicSqlService.dbRead(BasicSqlService.java:464)
         at 
org.sakaiproject.util.BaseDbFlatStorage.getSelectedResources(BaseDbFlatStorage.java:470)
         at 
org.sakaiproject.site.impl.DbSiteService$DbStorage.getSites(DbSiteService.java:782)
         at 
org.sakaiproject.site.impl.BaseSiteService.getSites(BaseSiteService.java:1616)
         at 
org.sakaiproject.site.cover.SiteService.getSites(SiteService.java:400)
         at 
org.sakaiproject.site.tool.MembershipAction.sizeResources(MembershipAction.java:85)
         at 
org.sakaiproject.cheftool.PagedResourceActionII.prepPage(PagedResourceActionII.java:270)
         at 
org.sakaiproject.site.tool.MembershipAction.buildMainPanelContext(MembershipAction.java:196)
         at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
         at 
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
         at 
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
         at java.lang.reflect.Method.invoke(Method.java:597)
         at 
org.sakaiproject.cheftool.VelocityPortletPaneledAction.toolModeDispatch(VelocityPortletPaneledAction.java:398)
         at 
org.sakaiproject.cheftool.ToolServlet.doGet(ToolServlet.java:236)
         at 
org.sakaiproject.cheftool.VelocityPortletPaneledAction.doGet(VelocityPortletPaneledAction.java:1029)
         at javax.servlet.http.HttpServlet.service(HttpServlet.java:627)
         at 
org.sakaiproject.vm.ComponentServlet.service(ComponentServlet.java:56)
         at javax.servlet.http.HttpServlet.service(HttpServlet.java:729)
         at 
org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:269)
         at 
org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:188)
         at 
org.sakaiproject.util.RequestFilter.doFilter(RequestFilter.java:580)
         at 
org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:215)
         at 
org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:188)
         at 
org.apache.catalina.core.ApplicationDispatcher.invoke(ApplicationDispatcher.java:659)
         at 
org.apache.catalina.core.ApplicationDispatcher.processRequest(ApplicationDispatcher.java:457)
         at 
org.apache.catalina.core.ApplicationDispatcher.doForward(ApplicationDispatcher.java:359)
         at 
org.apache.catalina.core.ApplicationDispatcher.forward(ApplicationDispatcher.java:311)
         at 
org.sakaiproject.tool.impl.ActiveToolComponent$MyActiveTool.forward(ActiveToolComponent.java:500)
         at 
org.sakaiproject.portal.charon.SkinnableCharonPortal.forwardTool(SkinnableCharonPortal.java:1216)
         at 
org.sakaiproject.portal.charon.handlers.ToolHandler.doTool(ToolHandler.java:204)
         at 
org.sakaiproject.portal.charon.handlers.ToolHandler.doGet(ToolHandler.java:96)
         at 
org.sakaiproject.portal.charon.SkinnableCharonPortal.doGet(SkinnableCharonPortal.java:768)


-- 
David Barroso Iglesias   david at asic.udl.cat
=====================
Universitat de Lleida
Àrea de sistemes d'Informació i Comunicacions
Cap de projectes interns

University of Lleida
Information and Communication Systems Service
Internal project manager

Tlf: +34 973 702040
Fax: +34 973 702130
=====================