[Building Sakai] Production Sakai 2.7.0 deadlocks
David Barroso
david at asic.udl.cat
Fri Aug 6 03:54:12 PDT 2010
Hi all,
Following the tuning recommendations given by Indiana and
Rutgers in the Denver conference, we have included two pool
parameters in sakai.properties
maxActive at javax.sql.BaseDataSource=150
timeBetweenEvictionRunsMillis at javax.sql.BaseDataSource=-1
And the problem disappeared.
Thanks a lot !!
By the way, the ArrayIndexOutOfBoundsException
on
sun.util.calendar.BaseCalendar.getCalendarDateFromFixedDate(BaseCalendar.java:436)
still remain when a high level of sessions appears.
It seems like a problem with the Calendar class, because
it is not thread-safe.
Regards,
On 08/04/2010 03:07 PM, David Haines wrote:
> If tuning / upgrading the dbcp pool implementation doesn't help then
> getting a thread dump when it is locked up would help. It would be
> good to know which threads are holding connections. It could also be
> useful to monitor memory usage to make sure that most of the time the
> JVM is running a significant amount of unused memory to reduce the
> frequency of GC.
>
> - Dave
>
> David Haines
> CTools Developer
> Digital Media Commons
> University of Michigan
> dlhaines at umich.edu <mailto:dlhaines at umich.edu>
>
>
>
> On Aug 4, 2010, at 6:50 AM, David Barroso wrote:
>
>> Hi all,
>>
>> We have a Sakai 2.7.0 instance in production using
>> Oracle 10g, Java1.6 and Linux.
>>
>> When a high level of sessions is reached, the
>> java garbage collector starts working and the
>> following error is shown in catalina.out
>>
>> Once the GregorSamsa class is unloaded and the
>> java.lang.IndexOutOfBoundsException appears,
>> the servers enters in a deadlock, waiting for
>> database connections
>>
>> org.apache.commons.dbcp.SQLNestedException: Cannot get a connection,
>> pool error Timeout waiting for idle object
>> Caused by: java.util.NoSuchElementException: Timeout waiting for idle
>> object
>>
>> We have found Oracle and Tomcat bug reports
>> talking about this, but in very old versions.
>>
>> Any idea or clue to solve this problem ?
>>
>> I've been watching the nightly2 oracle instance and
>> the Unloading class GregorSamsa appears at the end
>> of the catalina.out log.
>>
>> http://nightly2.sakaiproject.org/logs/tomcat-nightly-oracle/
>>
>> Regards,
>>
>>
>> -------------
>> our catalina.out
>>
>> [Unloading class GregorSamsa]
>> [Unloading class GregorSamsa$2]
>> [Unloading class GregorSamsa$0]
>> [Unloading class GregorSamsa]
>> [Unloading class GregorSamsa$4]
>> [Unloading class GregorSamsa$0]
>> [Unloading class GregorSamsa]
>> [Unloading class GregorSamsa]
>> [Unloading class GregorSamsa$2]
>> [Unloading class GregorSamsa$3]
>> [Unloading class GregorSamsa$4]
>> [Unloading class GregorSamsa$1]
>> 2010-08-04 12:17:07,075 WARN TP-Processor17
>> org.apache.myfaces.application.jsp.JspViewHandlerImpl - no faces servlet
>> mappings found
>> 2010-08-04 12:17:08,841 INFO TP-Processor10
>> org.apache.myfaces.shared_impl.config.MyfacesConfig - Tomahawk jar not
>> available. Autoscrolling, DetectJavascript, AddResourceClass and
>> CheckExtensionsFilter are disabled now.
>> 2010-08-04 12:17:08,841 INFO TP-Processor10
>> org.apache.myfaces.shared_impl.config.MyfacesConfig - Starting up
>> Tomahawk on the MyFaces-JSF-Implementation
>> 2010-08-04 12:17:12,435 ERROR http-8080-Processor61
>> org.apache.catalina.core.ContainerBase.[Catalina].[localhost].[/sakai-site-manage-tool].[sakai.membership]
>>
>> - Servlet.service() for servlet sakai.membership threw exception
>> java.lang.ArrayIndexOutOfBoundsException: -31
>> at
>> sun.util.calendar.BaseCalendar.getCalendarDateFromFixedDate(BaseCalendar.java:436)
>> at
>> java.util.GregorianCalendar.computeFields(GregorianCalendar.java:2081)
>> at
>> java.util.GregorianCalendar.computeTime(GregorianCalendar.java:2472)
>> at java.util.Calendar.updateTime(Calendar.java:2468)
>> at java.util.Calendar.getTimeInMillis(Calendar.java:1087)
>> at
>> oracle.jdbc.driver.DateTimeCommonAccessor.getTimestamp(DateTimeCommonAccessor.java:256)
>> at
>> oracle.jdbc.driver.OracleResultSetImpl.getTimestamp(OracleResultSetImpl.java:1644)
>> at
>> org.apache.commons.dbcp.DelegatingResultSet.getTimestamp(DelegatingResultSet.java:548)
>> at
>> org.sakaiproject.site.impl.DbSiteService$DbStorage.readSqlResultRecord(DbSiteService.java:2128)
>> at
>> org.sakaiproject.db.impl.BasicSqlService.dbRead(BasicSqlService.java:564)
>> at
>> org.sakaiproject.db.impl.BasicSqlService.dbRead(BasicSqlService.java:464)
>> at
>> org.sakaiproject.util.BaseDbFlatStorage.getSelectedResources(BaseDbFlatStorage.java:470)
>> at
>> org.sakaiproject.site.impl.DbSiteService$DbStorage.getSites(DbSiteService.java:782)
>> at
>> org.sakaiproject.site.impl.BaseSiteService.getSites(BaseSiteService.java:1616)
>> at
>> org.sakaiproject.site.cover.SiteService.getSites(SiteService.java:400)
>> at
>> org.sakaiproject.site.tool.MembershipAction.sizeResources(MembershipAction.java:85)
>> at
>> org.sakaiproject.cheftool.PagedResourceActionII.prepPage(PagedResourceActionII.java:270)
>> at
>> org.sakaiproject.site.tool.MembershipAction.buildMainPanelContext(MembershipAction.java:196)
>> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>> at
>> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
>> at
>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>> at java.lang.reflect.Method.invoke(Method.java:597)
>> at
>> org.sakaiproject.cheftool.VelocityPortletPaneledAction.toolModeDispatch(VelocityPortletPaneledAction.java:398)
>> at
>> org.sakaiproject.cheftool.ToolServlet.doGet(ToolServlet.java:236)
>> at
>> org.sakaiproject.cheftool.VelocityPortletPaneledAction.doGet(VelocityPortletPaneledAction.java:1029)
>> at javax.servlet.http.HttpServlet.service(HttpServlet.java:627)
>> at
>> org.sakaiproject.vm.ComponentServlet.service(ComponentServlet.java:56)
>> at javax.servlet.http.HttpServlet.service(HttpServlet.java:729)
>> at
>> org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:269)
>> at
>> org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:188)
>> at
>> org.sakaiproject.util.RequestFilter.doFilter(RequestFilter.java:580)
>> at
>> org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:215)
>> at
>> org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:188)
>> at
>> org.apache.catalina.core.ApplicationDispatcher.invoke(ApplicationDispatcher.java:659)
>> at
>> org.apache.catalina.core.ApplicationDispatcher.processRequest(ApplicationDispatcher.java:457)
>> at
>> org.apache.catalina.core.ApplicationDispatcher.doForward(ApplicationDispatcher.java:359)
>> at
>> org.apache.catalina.core.ApplicationDispatcher.forward(ApplicationDispatcher.java:311)
>> at
>> org.sakaiproject.tool.impl.ActiveToolComponent$MyActiveTool.forward(ActiveToolComponent.java:500)
>> at
>> org.sakaiproject.portal.charon.SkinnableCharonPortal.forwardTool(SkinnableCharonPortal.java:1216)
>> at
>> org.sakaiproject.portal.charon.handlers.ToolHandler.doTool(ToolHandler.java:204)
>> at
>> org.sakaiproject.portal.charon.handlers.ToolHandler.doGet(ToolHandler.java:96)
>> at
>> org.sakaiproject.portal.charon.SkinnableCharonPortal.doGet(SkinnableCharonPortal.java:768)
>>
>>
>> --
>> David Barroso Iglesias david at asic.udl.cat
>> =====================
>> Universitat de Lleida
>> Àrea de sistemes d'Informació i Comunicacions
>> Cap de projectes interns
>>
>> University of Lleida
>> Information and Communication Systems Service
>> Internal project manager
>>
>> Tlf: +34 973 702040
>> Fax: +34 973 702130
>> =====================
>>
>> _______________________________________________
>> sakai-dev mailing list
>> sakai-dev at collab.sakaiproject.org
>> http://collab.sakaiproject.org/mailman/listinfo/sakai-dev
>>
>> TO UNSUBSCRIBE: send email to
>> sakai-dev-unsubscribe at collab.sakaiproject.org with a subject of
>> "unsubscribe"
>>
>>
>
--
David Barroso Iglesias david at asic.udl.cat
=====================
Universitat de Lleida
Àrea de sistemes d'Informació i Comunicacions
Cap de projectes interns
University of Lleida
Information and Communication Systems Service
Internal project manager
Tlf: +34 973 702040
Fax: +34 973 702130
=====================
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://collab.sakaiproject.org/pipermail/sakai-dev/attachments/20100806/f965a667/attachment.html
More information about the sakai-dev
mailing list