[Building Sakai] Production Sakai 2.7.0 deadlocks

Wed Aug 4 06:07:16 PDT 2010

If tuning / upgrading the dbcp pool implementation doesn't help then getting a thread dump when it is locked up would help.  It would be good to know which threads are holding connections.  It could also be useful to monitor memory usage to make sure that most of the time the JVM is running a significant amount of unused memory to reduce the frequency of GC.

- Dave

David Haines
CTools Developer
Digital Media Commons
University of Michigan 
dlhaines at umich.edu

On Aug 4, 2010, at 6:50 AM, David Barroso wrote:

> Hi all,
> 
> We have a Sakai 2.7.0 instance in production using
> Oracle 10g, Java1.6 and Linux.
> 
> When a high level of sessions is reached, the
> java garbage collector starts working and the
> following error is shown in catalina.out
> 
> Once the GregorSamsa class is unloaded and the
> java.lang.IndexOutOfBoundsException appears,
> the servers enters in a deadlock, waiting for
> database connections
> 
> org.apache.commons.dbcp.SQLNestedException: Cannot get a connection, 
> pool error Timeout waiting for idle object
> Caused by: java.util.NoSuchElementException: Timeout waiting for idle object
> 
> We have found Oracle and Tomcat bug reports
> talking about this, but in very old versions.
> 
> Any idea or clue to solve this problem ?
> 
> I've been watching the nightly2 oracle instance and
> the Unloading class GregorSamsa appears at the end
> of the catalina.out log.
> 
> http://nightly2.sakaiproject.org/logs/tomcat-nightly-oracle/
> 
> Regards,
> 
> 
> -------------
> our catalina.out
> 
> [Unloading class GregorSamsa]
> [Unloading class GregorSamsa$2]
> [Unloading class GregorSamsa$0]
> [Unloading class GregorSamsa]
> [Unloading class GregorSamsa$4]
> [Unloading class GregorSamsa$0]
> [Unloading class GregorSamsa]
> [Unloading class GregorSamsa]
> [Unloading class GregorSamsa$2]
> [Unloading class GregorSamsa$3]
> [Unloading class GregorSamsa$4]
> [Unloading class GregorSamsa$1]
> 2010-08-04 12:17:07,075  WARN TP-Processor17 
> org.apache.myfaces.application.jsp.JspViewHandlerImpl - no faces servlet 
> mappings found
> 2010-08-04 12:17:08,841  INFO TP-Processor10 
> org.apache.myfaces.shared_impl.config.MyfacesConfig - Tomahawk jar not 
> available. Autoscrolling, DetectJavascript, AddResourceClass and 
> CheckExtensionsFilter are disabled now.
> 2010-08-04 12:17:08,841  INFO TP-Processor10 
> org.apache.myfaces.shared_impl.config.MyfacesConfig - Starting up 
> Tomahawk on the MyFaces-JSF-Implementation
> 2010-08-04 12:17:12,435 ERROR http-8080-Processor61 
> org.apache.catalina.core.ContainerBase.[Catalina].[localhost].[/sakai-site-manage-tool].[sakai.membership] 
> - Servlet.service() for servlet sakai.membership threw exception
> java.lang.ArrayIndexOutOfBoundsException: -31
>         at 
> sun.util.calendar.BaseCalendar.getCalendarDateFromFixedDate(BaseCalendar.java:436)
>         at 
> java.util.GregorianCalendar.computeFields(GregorianCalendar.java:2081)
>         at 
> java.util.GregorianCalendar.computeTime(GregorianCalendar.java:2472)
>         at java.util.Calendar.updateTime(Calendar.java:2468)
>         at java.util.Calendar.getTimeInMillis(Calendar.java:1087)
>         at 
> oracle.jdbc.driver.DateTimeCommonAccessor.getTimestamp(DateTimeCommonAccessor.java:256)
>         at 
> oracle.jdbc.driver.OracleResultSetImpl.getTimestamp(OracleResultSetImpl.java:1644)
>         at 
> org.apache.commons.dbcp.DelegatingResultSet.getTimestamp(DelegatingResultSet.java:548)
>         at 
> org.sakaiproject.site.impl.DbSiteService$DbStorage.readSqlResultRecord(DbSiteService.java:2128)
>         at 
> org.sakaiproject.db.impl.BasicSqlService.dbRead(BasicSqlService.java:564)
>         at 
> org.sakaiproject.db.impl.BasicSqlService.dbRead(BasicSqlService.java:464)
>         at 
> org.sakaiproject.util.BaseDbFlatStorage.getSelectedResources(BaseDbFlatStorage.java:470)
>         at 
> org.sakaiproject.site.impl.DbSiteService$DbStorage.getSites(DbSiteService.java:782)
>         at 
> org.sakaiproject.site.impl.BaseSiteService.getSites(BaseSiteService.java:1616)
>         at 
> org.sakaiproject.site.cover.SiteService.getSites(SiteService.java:400)
>         at 
> org.sakaiproject.site.tool.MembershipAction.sizeResources(MembershipAction.java:85)
>         at 
> org.sakaiproject.cheftool.PagedResourceActionII.prepPage(PagedResourceActionII.java:270)
>         at 
> org.sakaiproject.site.tool.MembershipAction.buildMainPanelContext(MembershipAction.java:196)
>         at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>         at 
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
>         at 
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>         at java.lang.reflect.Method.invoke(Method.java:597)
>         at 
> org.sakaiproject.cheftool.VelocityPortletPaneledAction.toolModeDispatch(VelocityPortletPaneledAction.java:398)
>         at 
> org.sakaiproject.cheftool.ToolServlet.doGet(ToolServlet.java:236)
>         at 
> org.sakaiproject.cheftool.VelocityPortletPaneledAction.doGet(VelocityPortletPaneledAction.java:1029)
>         at javax.servlet.http.HttpServlet.service(HttpServlet.java:627)
>         at 
> org.sakaiproject.vm.ComponentServlet.service(ComponentServlet.java:56)
>         at javax.servlet.http.HttpServlet.service(HttpServlet.java:729)
>         at 
> org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:269)
>         at 
> org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:188)
>         at 
> org.sakaiproject.util.RequestFilter.doFilter(RequestFilter.java:580)
>         at 
> org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:215)
>         at 
> org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:188)
>         at 
> org.apache.catalina.core.ApplicationDispatcher.invoke(ApplicationDispatcher.java:659)
>         at 
> org.apache.catalina.core.ApplicationDispatcher.processRequest(ApplicationDispatcher.java:457)
>         at 
> org.apache.catalina.core.ApplicationDispatcher.doForward(ApplicationDispatcher.java:359)
>         at 
> org.apache.catalina.core.ApplicationDispatcher.forward(ApplicationDispatcher.java:311)
>         at 
> org.sakaiproject.tool.impl.ActiveToolComponent$MyActiveTool.forward(ActiveToolComponent.java:500)
>         at 
> org.sakaiproject.portal.charon.SkinnableCharonPortal.forwardTool(SkinnableCharonPortal.java:1216)
>         at 
> org.sakaiproject.portal.charon.handlers.ToolHandler.doTool(ToolHandler.java:204)
>         at 
> org.sakaiproject.portal.charon.handlers.ToolHandler.doGet(ToolHandler.java:96)
>         at 
> org.sakaiproject.portal.charon.SkinnableCharonPortal.doGet(SkinnableCharonPortal.java:768)
> 
> 
> -- 
> David Barroso Iglesias   david at asic.udl.cat
> =====================
> Universitat de Lleida
> Àrea de sistemes d'Informació i Comunicacions
> Cap de projectes interns
> 
> University of Lleida
> Information and Communication Systems Service
> Internal project manager
> 
> Tlf: +34 973 702040
> Fax: +34 973 702130
> =====================
> 
> _______________________________________________
> sakai-dev mailing list
> sakai-dev at collab.sakaiproject.org
> http://collab.sakaiproject.org/mailman/listinfo/sakai-dev
> 
> TO UNSUBSCRIBE: send email to sakai-dev-unsubscribe at collab.sakaiproject.org with a subject of "unsubscribe"
> 
> 

-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://collab.sakaiproject.org/pipermail/sakai-dev/attachments/20100804/7606d631/attachment.html 