[Building Sakai] Production Sakai 2.7.0 deadlocks
Berg, Alan
A.M.Berg at uva.nl
Fri Aug 6 04:36:43 PDT 2010
Hi all,
The date class was spotted by static code review and Jiraed sometime back. It is also mentioned in confluence on the code quality secion by Aaron Z.
http://jira.sakaiproject.org/browse/SAK-13796
http://qa1-nl.sakaiproject.org/codereview/bug_dashboard/pmd_concurrency.html
The calendar tool was cleaned up, but rWiki and Reports still have some locations. If reports is stealthed then worth looking if you are getting traces from org.radeox.
Please Jira.
Alan B
Alan Berg
QA Director - The Sakai Foundation
Senior Developer / Quality Assurance
Group Education and Research Services
Central Computer Services
University of Amsterdam
http://home.uva.nl/a.m.berg
________________________________
From: sakai-dev-bounces at collab.sakaiproject.org [sakai-dev-bounces at collab.sakaiproject.org] on behalf of David Barroso [david at asic.udl.cat]
Sent: 06 August 2010 12:54
To: David Haines
Cc: sakai-dev
Subject: Re: [Building Sakai] Production Sakai 2.7.0 deadlocks
Hi all,
Following the tuning recommendations given by Indiana and
Rutgers in the Denver conference, we have included two pool
parameters in sakai.properties
maxActive at javax.sql.BaseDataSource=150<mailto:maxActive at javax.sql.BaseDataSource=150>
timeBetweenEvictionRunsMillis at javax.sql.BaseDataSource=-1<mailto:timeBetweenEvictionRunsMillis at javax.sql.BaseDataSource=-1>
And the problem disappeared.
Thanks a lot !!
By the way, the ArrayIndexOutOfBoundsException
on sun.util.calendar.BaseCalendar.getCalendarDateFromFixedDate(BaseCalendar.java:436)
still remain when a high level of sessions appears.
It seems like a problem with the Calendar class, because
it is not thread-safe.
Regards,
On 08/04/2010 03:07 PM, David Haines wrote:
If tuning / upgrading the dbcp pool implementation doesn't help then getting a thread dump when it is locked up would help. It would be good to know which threads are holding connections. It could also be useful to monitor memory usage to make sure that most of the time the JVM is running a significant amount of unused memory to reduce the frequency of GC.
- Dave
David Haines
CTools Developer
Digital Media Commons
University of Michigan
dlhaines at umich.edu<mailto:dlhaines at umich.edu>
On Aug 4, 2010, at 6:50 AM, David Barroso wrote:
Hi all,
We have a Sakai 2.7.0 instance in production using
Oracle 10g, Java1.6 and Linux.
When a high level of sessions is reached, the
java garbage collector starts working and the
following error is shown in catalina.out
Once the GregorSamsa class is unloaded and the
java.lang.IndexOutOfBoundsException appears,
the servers enters in a deadlock, waiting for
database connections
org.apache.commons.dbcp.SQLNestedException: Cannot get a connection,
pool error Timeout waiting for idle object
Caused by: java.util.NoSuchElementException: Timeout waiting for idle object
We have found Oracle and Tomcat bug reports
talking about this, but in very old versions.
Any idea or clue to solve this problem ?
I've been watching the nightly2 oracle instance and
the Unloading class GregorSamsa appears at the end
of the catalina.out log.
http://nightly2.sakaiproject.org/logs/tomcat-nightly-oracle/
Regards,
-------------
our catalina.out
[Unloading class GregorSamsa]
[Unloading class GregorSamsa$2]
[Unloading class GregorSamsa$0]
[Unloading class GregorSamsa]
[Unloading class GregorSamsa$4]
[Unloading class GregorSamsa$0]
[Unloading class GregorSamsa]
[Unloading class GregorSamsa]
[Unloading class GregorSamsa$2]
[Unloading class GregorSamsa$3]
[Unloading class GregorSamsa$4]
[Unloading class GregorSamsa$1]
2010-08-04 12:17:07,075 WARN TP-Processor17
org.apache.myfaces.application.jsp.JspViewHandlerImpl - no faces servlet
mappings found
2010-08-04 12:17:08,841 INFO TP-Processor10
org.apache.myfaces.shared_impl.config.MyfacesConfig - Tomahawk jar not
available. Autoscrolling, DetectJavascript, AddResourceClass and
CheckExtensionsFilter are disabled now.
2010-08-04 12:17:08,841 INFO TP-Processor10
org.apache.myfaces.shared_impl.config.MyfacesConfig - Starting up
Tomahawk on the MyFaces-JSF-Implementation
2010-08-04 12:17:12,435 ERROR http-8080-Processor61
org.apache.catalina.core.ContainerBase.[Catalina].[localhost].[/sakai-site-manage-tool].[sakai.membership]
- Servlet.service() for servlet sakai.membership threw exception
java.lang.ArrayIndexOutOfBoundsException: -31
at
sun.util.calendar.BaseCalendar.getCalendarDateFromFixedDate(BaseCalendar.java:436)
at
java.util.GregorianCalendar.computeFields(GregorianCalendar.java:2081)
at
java.util.GregorianCalendar.computeTime(GregorianCalendar.java:2472)
at java.util.Calendar.updateTime(Calendar.java:2468)
at java.util.Calendar.getTimeInMillis(Calendar.java:1087)
at
oracle.jdbc.driver.DateTimeCommonAccessor.getTimestamp(DateTimeCommonAccessor.java:256)
at
oracle.jdbc.driver.OracleResultSetImpl.getTimestamp(OracleResultSetImpl.java:1644)
at
org.apache.commons.dbcp.DelegatingResultSet.getTimestamp(DelegatingResultSet.java:548)
at
org.sakaiproject.site.impl.DbSiteService$DbStorage.readSqlResultRecord(DbSiteService.java:2128)
at
org.sakaiproject.db.impl.BasicSqlService.dbRead(BasicSqlService.java:564)
at
org.sakaiproject.db.impl.BasicSqlService.dbRead(BasicSqlService.java:464)
at
org.sakaiproject.util.BaseDbFlatStorage.getSelectedResources(BaseDbFlatStorage.java:470)
at
org.sakaiproject.site.impl.DbSiteService$DbStorage.getSites(DbSiteService.java:782)
at
org.sakaiproject.site.impl.BaseSiteService.getSites(BaseSiteService.java:1616)
at
org.sakaiproject.site.cover.SiteService.getSites(SiteService.java:400)
at
org.sakaiproject.site.tool.MembershipAction.sizeResources(MembershipAction.java:85)
at
org.sakaiproject.cheftool.PagedResourceActionII.prepPage(PagedResourceActionII.java:270)
at
org.sakaiproject.site.tool.MembershipAction.buildMainPanelContext(MembershipAction.java:196)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at
org.sakaiproject.cheftool.VelocityPortletPaneledAction.toolModeDispatch(VelocityPortletPaneledAction.java:398)
at
org.sakaiproject.cheftool.ToolServlet.doGet(ToolServlet.java:236)
at
org.sakaiproject.cheftool.VelocityPortletPaneledAction.doGet(VelocityPortletPaneledAction.java:1029)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:627)
at
org.sakaiproject.vm.ComponentServlet.service(ComponentServlet.java:56)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:729)
at
org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:269)
at
org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:188)
at
org.sakaiproject.util.RequestFilter.doFilter(RequestFilter.java:580)
at
org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:215)
at
org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:188)
at
org.apache.catalina.core.ApplicationDispatcher.invoke(ApplicationDispatcher.java:659)
at
org.apache.catalina.core.ApplicationDispatcher.processRequest(ApplicationDispatcher.java:457)
at
org.apache.catalina.core.ApplicationDispatcher.doForward(ApplicationDispatcher.java:359)
at
org.apache.catalina.core.ApplicationDispatcher.forward(ApplicationDispatcher.java:311)
at
org.sakaiproject.tool.impl.ActiveToolComponent$MyActiveTool.forward(ActiveToolComponent.java:500)
at
org.sakaiproject.portal.charon.SkinnableCharonPortal.forwardTool(SkinnableCharonPortal.java:1216)
at
org.sakaiproject.portal.charon.handlers.ToolHandler.doTool(ToolHandler.java:204)
at
org.sakaiproject.portal.charon.handlers.ToolHandler.doGet(ToolHandler.java:96)
at
org.sakaiproject.portal.charon.SkinnableCharonPortal.doGet(SkinnableCharonPortal.java:768)
--
David Barroso Iglesias david at asic.udl.cat
=====================
Universitat de Lleida
Àrea de sistemes d'Informació i Comunicacions
Cap de projectes interns
University of Lleida
Information and Communication Systems Service
Internal project manager
Tlf: +34 973 702040
Fax: +34 973 702130
=====================
_______________________________________________
sakai-dev mailing list
sakai-dev at collab.sakaiproject.org<mailto:sakai-dev at collab.sakaiproject.org>
http://collab.sakaiproject.org/mailman/listinfo/sakai-dev
TO UNSUBSCRIBE: send email to sakai-dev-unsubscribe at collab.sakaiproject.org<mailto:sakai-dev-unsubscribe at collab.sakaiproject.org> with a subject of "unsubscribe"
--
David Barroso Iglesias david at asic.udl.cat
=====================
Universitat de Lleida
Àrea de sistemes d'Informació i Comunicacions
Cap de projectes interns
University of Lleida
Information and Communication Systems Service
Internal project manager
Tlf: +34 973 702040
Fax: +34 973 702130
=====================
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://collab.sakaiproject.org/pipermail/sakai-dev/attachments/20100806/118c1bb9/attachment.html
More information about the sakai-dev
mailing list