[Building Sakai] MSGCNTR: unable to load and post forums

Kenwrick Chan kchan at hawaii.edu
Tue Oct 9 14:37:55 PDT 2012 

We didn't try 2.7.0, but the patch in msgcntr-575 is pretty simple and
direct, applying for mysql users only.  Not sure if would work for 2.7.0
but I think it's worth a try on a test system.  We missed it on one of our
front ends and the issue didn't consistently creep up and could go weeks
without.

https://jira.sakaiproject.org/browse/MSGCNTR-575

kenwrick

On Tue, Oct 9, 2012 at 11:17 AM, Sanghyun Jeon <euksa99 at gmail.com> wrote:

> Thank you for your reply.
> Yes, our DB is hammered badly. we need to kill the query every 5 minutes.
>
> Since it takes time to upgrade MSGCNTR version, is there any emergency
> patch or restarting tomcat, or other option to relieve this issue, we can
> use right now?
>
> Any suggestion!
>
> On Tue, Oct 9, 2012 at 1:54 PM, Kenwrick Chan <kchan at hawaii.edu> wrote:
>
>> Is the db getting hammered during this time?
>>
>> Do you also find something similar is the slow query logs?  In any case
>> this thread appears to support the claim that you should consider not using
>> msgcntr 2.7.0 (we're using 2.7.4)
>>
>> http://sakai-project-mail-list-archives.1343168.n2.nabble.com/Building-Sakai-MSGCNTR-Forums-mysql-picks-a-slow-query-plan-over-a-faster-query-plan-td6755140.html
>>
>> but since you're using mysql I suspect that you may be hit by this one,
>> with the query occasionally getting stuck on the mysql server side of the
>> house spinning away and never resolving, eventually consuming all resources.
>>
>> https://jira.sakaiproject.org/browse/MSGCNTR-575
>>
>>  On Tue, Oct 9, 2012 at 10:37 AM, Sanghyun Jeon <euksa99 at gmail.com>wrote:
>>
>>>  We, Claremont Consortium, are currently experiencing a lot of error
>>> logs related with Message & Forums tool. Users are complaining about the
>>> forum loading issues and they are unable to post the forum.  While
>>> monitoring Tomcat error log, we found the query it was trying to perform:
>>>
>>>  shows select count(*) as col_0_0_ from MFR_TOPIC_T topicimpl0_ inner
>>> join MFR_OPEN_FORUM_T openforumi2_ on
>>> topicimpl0_.of_surrogateKey=openforumi2_.ID inner join MFR_AREA_T
>>> areaimpl3_ on openforumi2_.surrogateKey=areaimpl3_.ID inner join
>>> MFR_MEMBERSHIP_ITEM_T membership4_ on
>>> topicimpl0_.ID=membership4_.t_surrogateKey, MFR_PERMISSION_LEVEL_T
>>> permission1_ where areaimpl3_.CONTEXT_ID=? and topicimpl0_.MODERATED=1 and
>>> (membership4_.NAME in (?)) and permission1_.MODERATE_POSTINGS=1 and
>>> (permission1_.TYPE_UUID<>? and
>>> permission1_.NAME=membership4_.PERMISSION_LEVEL_NAME or
>>> membership4_.PERMISSION_LEVEL=permission1_.ID)
>>>
>>>
>>>
>>> We thought we found several related JIRAs, such as MSGCNTR-364, but our
>>> db already has Context_id  index on MFR_AREA_T
>>>
>>> While we check our tables, we found the below information related with
>>> the query:
>>>
>>>
>>>
>>> MFR_AREA_T(23,804 records)
>>> MFR_OPEN_FORUM_T(12,422 records)
>>>
>>> MFR_PERMISSION_LEVEL_T(65,403 records)
>>> MFR_MEMBERSHIP_ITEM_T (130,594 records)
>>> MFR_TOPIC_T( 346,603 records), but 114 records where MODERATED=1
>>>
>>>
>>>
>>> We are using
>>>
>>> OS: CentOS *2.6.32-279.2.1.el6.x86_64*
>>>
>>> Hibernate version *3.2.7.ga *
>>>
>>> java version "1.5.0_18"
>>>
>>> Java(TM) 2 Runtime Environment, Standard Edition (build 1.5.0_18-b02)
>>>
>>> Java HotSpot(TM) 64-Bit Server VM (build 1.5.0_18-b02, mixed mode)
>>>
>>> Tomcat 5.5.28
>>>
>>> MySQL  5.1.61
>>>
>>> Sakai 2.7.0 with MSGCNTR 2.7.0
>>>
>>>
>>>
>>>
>>>
>>> While Forum tool is tried to load the page and fails to load it, it
>>> throws the below error for a user. Any insight to relieve this issue will
>>> be greatly appreciated. Thank you.
>>>
>>>
>>>
>>> Sanghyun Jeon, Ph.D
>>>
>>> Information Technology Services
>>>
>>> Pomona College
>>>
>>>
>>>
>>>
>>>
>>> org.sakaiproject.portal.api.PortalHandlerException:
>>> org.sakaiproject.tool.api.ToolException: javax.servlet.jsp.JspException:
>>> javax.faces.el.EvaluationException: Error getting property
>>> 'displayPendingMsgQueue' from bean of type
>>> org.sakaiproject.tool.messageforums.DiscussionForumTool:
>>> org.springframework.transaction.TransactionSystemException: Could not roll
>>> back Hibernate transaction; nested exception is
>>> org.hibernate.TransactionException: JDBC rollback failed
>>>
>>>     at
>>> org.sakaiproject.portal.charon.SkinnableCharonPortal.doGet(SkinnableCharonPortal.java:768)
>>>
>>> caused by: org.sakaiproject.tool.api.ToolException:
>>> javax.servlet.jsp.JspException: javax.faces.el.EvaluationException: Error
>>> getting property 'displayPendingMsgQueue' from bean of type
>>> org.sakaiproject.tool.messageforums.DiscussionForumTool:
>>> org.springframework.transaction.TransactionSystemException: Could not roll
>>> back Hibernate transaction; nested exception is
>>> org.hibernate.TransactionException: JDBC rollback failed
>>>
>>>     at
>>> org.sakaiproject.portal.charon.SkinnableCharonPortal.forwardTool(SkinnableCharonPortal.java:1216)
>>>
>>> caused by: org.apache.jasper.JasperException:
>>> javax.servlet.jsp.JspException: javax.faces.el.EvaluationException: Error
>>> getting property 'displayPendingMsgQueue' from bean of type
>>> org.sakaiproject.tool.messageforums.DiscussionForumTool:
>>> org.springframework.transaction.TransactionSystemException: Could not roll
>>> back Hibernate transaction; nested exception is
>>> org.hibernate.TransactionException: JDBC rollback failed
>>>
>>>     at
>>> org.apache.jasper.servlet.JspServletWrapper.service(JspServletWrapper.java:355)
>>>
>>> caused by: javax.servlet.ServletException:
>>> javax.servlet.jsp.JspException: javax.faces.el.EvaluationException: Error
>>> getting property 'displayPendingMsgQueue' from bean of type
>>> org.sakaiproject.tool.messageforums.DiscussionForumTool:
>>> org.springframework.transaction.TransactionSystemException: Could not roll
>>> back Hibernate transaction; nested exception is
>>> org.hibernate.TransactionException: JDBC rollback failed
>>>
>>>     at
>>> org.apache.jasper.runtime.PageContextImpl.handlePageException(PageContextImpl.java:774)
>>>
>>> caused by: javax.faces.el.EvaluationException:
>>> javax.faces.el.EvaluationException: Error getting property
>>> 'displayPendingMsgQueue' from bean of type
>>> org.sakaiproject.tool.messageforums.DiscussionForumTool:
>>> org.springframework.transaction.TransactionSystemException: Could not roll
>>> back Hibernate transaction; nested exception is
>>> org.hibernate.TransactionException: JDBC rollback failed
>>>
>>>     at
>>> com.sun.faces.el.ValueBindingImpl.getValue(ValueBindingImpl.java:154)
>>>
>>> caused by: javax.faces.el.EvaluationException: Error getting property
>>> 'displayPendingMsgQueue' from bean of type
>>> org.sakaiproject.tool.messageforums.DiscussionForumTool:
>>> org.springframework.transaction.TransactionSystemException: Could not roll
>>> back Hibernate transaction; nested exception is
>>> org.hibernate.TransactionException: JDBC rollback failed
>>>
>>>     at com.sun.faces.el.impl.ArraySuffix.evaluate(ArraySuffix.java:167)
>>>
>>> caused by: org.springframework.transaction.TransactionSystemException:
>>> Could not roll back Hibernate transaction; nested exception is
>>> org.hibernate.TransactionException: JDBC rollback failed
>>>
>>>     at
>>> org.springframework.transaction.support.AbstractPlatformTransactionManager.processRollback(AbstractPlatformTransactionManager.java:823)
>>>
>>> caused by: org.hibernate.TransactionException: JDBC rollback failed
>>>
>>>     at
>>> org.springframework.orm.hibernate3.HibernateTransactionManager.doRollback(HibernateTransactionManager.java:674)
>>>
>>> caused by:
>>> com.mysql.jdbc.exceptions.MySQLNonTransientConnectionException:
>>> Communications link failure during rollback(). Transaction resolution
>>> unknown.
>>>
>>>     at com.mysql.jdbc.SQLError.createSQLException(SQLError.java:1011)
>>>
>>>     at com.mysql.jdbc.SQLError.createSQLException(SQLError.java:988)
>>>
>>>     at com.mysql.jdbc.SQLError.createSQLException(SQLError.java:974)
>>>
>>>     at com.mysql.jdbc.SQLError.createSQLException(SQLError.java:919)
>>>
>>>     at com.mysql.jdbc.ConnectionImpl.rollback(ConnectionImpl.java:4793)
>>>
>>>     at
>>> org.apache.commons.dbcp.DelegatingConnection.rollback(DelegatingConnection.java:328)
>>>
>>>     at
>>> org.apache.commons.dbcp.PoolingDataSource$PoolGuardConnectionWrapper.rollback(PoolingDataSource.java:312)
>>>
>>>     at sun.reflect.GeneratedMethodAccessor28884.invoke(null:-1)
>>>
>>>     at
>>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>>>
>>>     at java.lang.reflect.Method.invoke(Method.java:585)
>>>
>>>     at
>>> org.springframework.jdbc.datasource.LazyConnectionDataSourceProxy$LazyConnectionInvocationHandler.invoke(LazyConnectionDataSourceProxy.java:364)
>>>
>>>     at $Proxy0.rollback(null:-1)
>>>
>>>     at
>>> org.hibernate.transaction.JDBCTransaction.rollbackAndResetAutoCommit(JDBCTransaction.java:183)
>>>
>>>     at
>>> org.hibernate.transaction.JDBCTransaction.rollback(JDBCTransaction.java:162)
>>>
>>>     at
>>> org.springframework.orm.hibernate3.HibernateTransactionManager.doRollback(HibernateTransactionManager.java:674)
>>>
>>>     at
>>> org.springframework.transaction.support.AbstractPlatformTransactionManager.processRollback(AbstractPlatformTransactionManager.java:823)
>>>
>>>     at
>>> org.springframework.transaction.support.AbstractPlatformTransactionManager.rollback(AbstractPlatformTransactionManager.java:800)
>>>
>>>     at
>>> org.springframework.transaction.interceptor.TransactionAspectSupport.completeTransactionAfterThrowing(TransactionAspectSupport.java:339)
>>>
>>>     at
>>> org.springframework.transaction.interceptor.TransactionInterceptor.invoke(TransactionInterceptor.java:110)
>>>
>>>     at
>>> org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:171)
>>>
>>>     at
>>> org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:204)
>>>
>>>     at $Proxy19.getNumModTopicsWithModPermission(null:-1)
>>>
>>>     at
>>> org.sakaiproject.tool.messageforums.DiscussionForumTool.isDisplayPendingMsgQueue(DiscussionForumTool.java:4607)
>>>
>>>     at sun.reflect.GeneratedMethodAccessor1922.invoke(null:-1)
>>>
>>>     at
>>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>>>
>>>     at java.lang.reflect.Method.invoke(Method.java:585)
>>>
>>>     at
>>> com.sun.faces.el.PropertyResolverImpl.getValue(PropertyResolverImpl.java:79)
>>>
>>>     at com.sun.faces.el.impl.ArraySuffix.evaluate(ArraySuffix.java:167)
>>>
>>>     at com.sun.faces.el.impl.ComplexValue.evaluate(ComplexValue.java:151)
>>>
>>>     at
>>> com.sun.faces.el.impl.ExpressionEvaluatorImpl.evaluate(ExpressionEvaluatorImpl.java:243)
>>>
>>>     at
>>> com.sun.faces.el.ValueBindingImpl.getValue(ValueBindingImpl.java:173)
>>>
>>>     at
>>> com.sun.faces.el.ValueBindingImpl.getValue(ValueBindingImpl.java:154)
>>>
>>>     at
>>> javax.faces.component.UIComponentBase.isRendered(UIComponentBase.java:315)
>>>
>>>     at
>>> javax.faces.webapp.UIComponentTag.shouldBeSuppressed(UIComponentTag.java:1281)
>>>
>>>     at
>>> javax.faces.webapp.UIComponentTag.doStartTag(UIComponentTag.java:475)
>>>
>>>     at
>>> com.sun.faces.taglib.html_basic.CommandButtonTag.doStartTag(CommandButtonTag.java:
>>> at
>>> org.apache.jsp.jsp.discussionForum.forumsOnly.dfForums_jsp._jspx_meth_sakai_005ftool_005fbar_005fitem_005f4(dfForums_jsp.java:510)
>>>
>>>     at
>>> org.apache.jsp.jsp.discussionForum.forumsOnly.dfForums_jsp._jspx_meth_sakai_005ftool_005fbar_005f0(dfForums_jsp.java:401)
>>>
>>>     at
>>> org.apache.jsp.jsp.discussionForum.forumsOnly.dfForums_jsp._jspService(dfForums_jsp.java:252)
>>>
>>>     at org.apache.jasper.runtime.HttpJspBase.service(HttpJspBase.java:98)
>>>
>>>     at javax.servlet.http.HttpServlet.service(HttpServlet.java:729)
>>>
>>>     at
>>> org.apache.jasper.servlet.JspServletWrapper.service(JspServletWrapper.java:331)
>>>
>>>     at
>>> org.apache.jasper.servlet.JspServlet.serviceJspFile(JspServlet.java:329)
>>>
>>>     at org.apache.jasper.servlet.JspServlet.service(JspServlet.java:265)
>>>
>>>     at javax.servlet.http.HttpServlet.service(HttpServlet.java:729)
>>>
>>>     at
>>> org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:269)
>>>
>>>     at
>>> org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:188)
>>>
>>>     at
>>> org.apache.catalina.core.ApplicationDispatcher.invoke(ApplicationDispatcher.java:679)
>>>
>>>     at
>>> org.apache.catalina.core.ApplicationDispatcher.processRequest(ApplicationDispatcher.java:461)
>>>
>>>     at
>>> org.apache.catalina.core.ApplicationDispatcher.doForward(ApplicationDispatcher.java:399)
>>>
>>>     at
>>> org.apache.catalina.core.ApplicationDispatcher.forward(ApplicationDispatcher.java:301)
>>>
>>>     at
>>> com.sun.faces.context.ExternalContextImpl.dispatch(ExternalContextImpl.java:322)
>>>
>>>     at
>>> com.sun.faces.application.ViewHandlerImpl.renderView(ViewHandlerImpl.java:130)
>>>
>>>     at
>>> org.sakaiproject.jsf.app.SakaiViewHandler.renderView(SakaiViewHandler.java:137)
>>>
>>>     at
>>> com.sun.faces.lifecycle.RenderResponsePhase.execute(RenderResponsePhase.java:87)
>>>
>>>     at
>>> com.sun.faces.lifecycle.LifecycleImpl.phase(LifecycleImpl.java:200)
>>>
>>>     at
>>> com.sun.faces.lifecycle.LifecycleImpl.render(LifecycleImpl.java:117)
>>>
>>>     at javax.faces.webapp.FacesServlet.service(FacesServlet.java:198)
>>>
>>>     at
>>> org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:269)
>>>
>>>     at
>>> org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:188)
>>>
>>>     at
>>> org.sakaiproject.util.RequestFilter.doFilter(RequestFilter.java:580)
>>>
>>>     at
>>> org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:215)
>>>
>>>     at
>>> org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:188)
>>>
>>>     at
>>> org.apache.catalina.core.ApplicationDispatcher.invoke(ApplicationDispatcher.java:679)
>>>
>>>     at
>>> org.apache.catalina.core.ApplicationDispatcher.processRequest(ApplicationDispatcher.java:461)
>>>
>>>     at
>>> org.apache.catalina.core.ApplicationDispatcher.doForward(ApplicationDispatcher.java:399)
>>>
>>>     at
>>> org.apache.catalina.core.ApplicationDispatcher.forward(ApplicationDispatcher.java:301)
>>>
>>>     at
>>> org.sakaiproject.tool.messageforums.MessageForumsFilePickerServlet.dispatch(MessageForumsFilePickerServlet.java:166)
>>>
>>>     at org.sakaiproject.jsf.util.JsfTool.doGet(JsfTool.java:241)
>>>
>>>     at javax.servlet.http.HttpServlet.service(HttpServlet.java:627)
>>>
>>>     at javax.servlet.http.HttpServlet.service(HttpServlet.java:729)
>>>
>>>     at
>>> org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:269)
>>>
>>>     at
>>> org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:188)
>>>
>>>     at
>>> org.sakaiproject.util.RequestFilter.doFilter(RequestFilter.java:580)
>>>
>>>     at
>>> org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:215)
>>>
>>>     at
>>> org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:188)
>>>
>>>     at
>>> org.apache.catalina.core.ApplicationDispatcher.invoke(ApplicationDispatcher.java:679)
>>>
>>>     at
>>> org.apache.catalina.core.ApplicationDispatcher.processRequest(ApplicationDispatcher.java:461)
>>>
>>>     at
>>> org.apache.catalina.core.ApplicationDispatcher.doForward(ApplicationDispatcher.java:364)
>>>
>>>     at
>>> org.apache.catalina.core.ApplicationDispatcher.forward(ApplicationDispatcher.java:301)
>>>
>>>     at
>>> org.sakaiproject.tool.impl.ActiveToolComponent$MyActiveTool.forward(ActiveToolComponent.java:500)
>>>
>>>     at
>>> org.sakaiproject.portal.charon.SkinnableCharonPortal.forwardTool(SkinnableCharonPortal.java:1216)
>>>
>>>     at
>>> org.sakaiproject.portal.charon.handlers.ToolHandler.doTool(ToolHandler.java:204)
>>>
>>>     at
>>> org.sakaiproject.portal.charon.handlers.ToolHandler.doGet(ToolHandler.java:96)
>>>
>>>     at
>>> org.sakaiproject.portal.charon.SkinnableCharonPortal.doGet(SkinnableCharonPortal.java:768)
>>>
>>>     at javax.servlet.http.HttpServlet.service(HttpServlet.java:627)
>>>
>>>     at javax.servlet.http.HttpServlet.service(HttpServlet.java:729)
>>>
>>>     at
>>> org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:269)
>>>
>>>     at
>>> org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:188)
>>>
>>>     at
>>> org.sakaiproject.util.RequestFilter.doFilter(RequestFilter.java:641)
>>>
>>>     at
>>> org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:215)
>>>
>>>     at
>>> org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:188)
>>>
>>>     at
>>> org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:213)
>>>
>>>     at
>>> org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:172)
>>>
>>>     at
>>> org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:127)
>>>
>>>     at
>>> org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:117)
>>>
>>>     at
>>> org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:108)
>>>
>>>     at
>>> org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:174)
>>>
>>>     at
>>> org.apache.coyote.http11.Http11Processor.process(Http11Processor.java:873)
>>>
>>>     at
>>> org.apache.coyote.http11.Http11BaseProtocol$Http11ConnectionHandler.processConnection(Http11BaseProtocol.java:665)
>>>
>>>     at
>>> org.apache.tomcat.util.net.PoolTcpEndpoint.processSocket(PoolTcpEndpoint.java:528)
>>>
>>>     at
>>> org.apache.tomcat.util.net.LeaderFollowerWorkerThread.runIt(LeaderFollowerWorkerThread.java:81)
>>>
>>>     at
>>> org.apache.tomcat.util.threads.ThreadPool$ControlRunnable.run(ThreadPool.java:689)
>>>
>>>     at java.lang.Thread.run(Thread.java:595)
>>>
>>>
>>>
>>>
>>>
>>> user: 43a13170-d002-47d1-af5c-d851afe54830
>>>
>>>
>>>
>>> usage-session: 54b4e368-94f9-479a-837c-4f905c4ed175
>>>
>>>
>>>
>>> time: Oct 9, 2012 12:30:01
>>>
>>>
>>>
>>>
>>>
>>> _______________________________________________
>>> sakai-dev mailing list
>>> sakai-dev at collab.sakaiproject.org
>>> http://collab.sakaiproject.org/mailman/listinfo/sakai-dev
>>>
>>> TO UNSUBSCRIBE: send email to
>>> sakai-dev-unsubscribe at collab.sakaiproject.org with a subject of
>>> "unsubscribe"
>>>
>>
>>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://collab.sakaiproject.org/pipermail/sakai-dev/attachments/20121009/a90beb1d/attachment.html

More information about the sakai-dev mailing list