[Building Sakai] Application Servers Crashing
Warwick Chapman
warwickchapman at gmail.com
Fri Nov 1 01:04:40 PDT 2013
Hi All
I'm going mad here. Surely we can expect a four processor, 4G appserver to
handle more than 50 sessions without crashing?
Mine keep crashing and its causing havoc.
As they approach 40 sessions, the system load which is usually below 0.10
start climbing, and then hits 1.00 and eventually I get the
"java.lang.OutOfMemoryError: Java heap space" error.
*Details*
1. Ubuntu 12.04 OpenVZ on Proxmox with 4G RAM and 4 processors allocated.
2. Oracle Java 6:
root at opportunity-10:~# java -version
java version "1.6.0_45"
Java(TM) SE Runtime Environment (build 1.6.0_45-b06)
Java HotSpot(TM) 64-Bit Server VM (build 20.45-b01, mixed mode)
3. Environment variables:
root at opportunity-10:~# set | grep JAVA
JAVA_HOME=/usr/lib/jvm/java-6-oracle
JAVA_OPTS='-server -Xms512m -Xmx1024m -XX:PermSize=128m
-XX:MaxPermSize=512m -XX:NewSize=192m -XX:MaxNewSize=384m
-Djava.awt.headless=true -Dhttp.agent=Sakai
-Dorg.apache.jasper.compiler.Parser.STRICT_QUOTE_ESCAPING=false
-Dsun.lang.ClassLoader.allowArraySyntax=true -Duser.language=en
-Duser.region=ZA -Dfile.encoding=UTF-8'
4. Sakai 2.9.3 built independently on each app server (also tried building
on one and deploying to others). Custom providers, SMS and only patch is
for UCT SMS.
5. Process:
31032 pts/2 Sl 1:44 /usr/lib/jvm/java-6-oracle/bin/java
-Djava.util.logging.config.file=/opt/tomcat/conf/logging.properties
-Djava.util.logging.manager=org.apache.juli.ClassLoaderLogManager -server
-Xms512m -Xmx1024m -XX:PermSize=128m -XX:MaxPermSize=512m -XX:NewSize=192m
-XX:MaxNewSize=384m -Djava.awt.headless=true -Dhttp.agent=Sakai
-Dorg.apache.jasper.compiler.Parser.STRICT_QUOTE_ESCAPING=false
-Dsun.lang.ClassLoader.allowArraySyntax=true -Duser.language=en
-Duser.region=ZA -Dfile.encoding=UTF-8
-Djava.endorsed.dirs=/opt/tomcat/endorsed -classpath
/opt/tomcat/bin/bootstrap.jar:/opt/tomcat/bin/tomcat-juli.jar
-Dcatalina.base=/opt/tomcat -Dcatalina.home=/opt/tomcat
-Djava.io.tmpdir=/opt/tomcat/temp org.apache.catalina.startup.Bootstrap
start
6. Database server and LDAP server are quiet and not battling with load at
all.
7. The underlying host machines are quiet and load is not affecting them at
all and memory usage is safely below available.
See catalina for the most recent crash:
2013-11-01 09:37:18,497 WARN http-bio-8080-exec-281
org.sakaiproject.portal.charon.site.DefaultSiteViewImpl - Failed to set
canAddSite for current user. Defaulting to false ...
Exception in thread "PageSavingThread-sakai.profile2"
java.lang.OutOfMemoryError: Java heap space
2013-11-01 09:41:04,553 ERROR http-bio-8080-exec-256
org.hibernate.transaction.JDBCTransaction - JDBC rollback failed
com.mysql.jdbc.exceptions.jdbc4.MySQLNonTransientConnectionException: No
operations allowed after connection closed.
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native
Method)
at
sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:39)
at
sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:27)
at java.lang.reflect.Constructor.newInstance(Constructor.java:513)
at com.mysql.jdbc.Util.handleNewInstance(Util.java:411)
at com.mysql.jdbc.Util.getInstance(Util.java:386)
at com.mysql.jdbc.SQLError.createSQLException(SQLError.java:1015)
at com.mysql.jdbc.SQLError.createSQLException(SQLError.java:989)
at com.mysql.jdbc.SQLError.createSQLException(SQLError.java:975)
at com.mysql.jdbc.SQLError.createSQLException(SQLError.java:920)
at
com.mysql.jdbc.ConnectionImpl.throwConnectionClosedException(ConnectionImpl.java:1304)
at
com.mysql.jdbc.ConnectionImpl.checkClosed(ConnectionImpl.java:1296)
at com.mysql.jdbc.ConnectionImpl.rollback(ConnectionImpl.java:5028)
at
org.apache.commons.dbcp.DelegatingConnection.rollback(DelegatingConnection.java:328)
at
org.apache.commons.dbcp.PoolingDataSource$PoolGuardConnectionWrapper.rollback(PoolingDataSource.java:312)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at
org.springframework.jdbc.datasource.LazyConnectionDataSourceProxy$LazyConnectionInvocationHandler.invoke(LazyConnectionDataSourceProxy.java:364)
at com.sun.proxy.$Proxy5.rollback(Unknown Source)
at
org.hibernate.transaction.JDBCTransaction.rollbackAndResetAutoCommit(JDBCTransaction.java:183)
at
org.hibernate.transaction.JDBCTransaction.rollback(JDBCTransaction.java:162)
at
org.springframework.orm.hibernate3.HibernateTransactionManager.doRollback(HibernateTransactionManager.java:674)
at
org.springframework.transaction.support.AbstractPlatformTransactionManager.processRollback(AbstractPlatformTransactionManager.java:823)
at
org.springframework.transaction.support.AbstractPlatformTransactionManager.rollback(AbstractPlatformTransactionManager.java:800)
at
org.springframework.transaction.interceptor.TransactionAspectSupport.completeTransactionAfterThrowing(TransactionAspectSupport.java:339)
at
org.springframework.transaction.interceptor.TransactionInterceptor.invoke(TransactionInterceptor.java:110)
at
org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:171)
at
org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:204)
at com.sun.proxy.$Proxy42.getPublishedAssessment(Unknown Source)
at
org.sakaiproject.tool.assessment.services.assessment.PublishedAssessmentService.getPublishedAssessment(PublishedAssessmentService.java:189)
at
org.sakaiproject.tool.assessment.ui.listener.delivery.BeginDeliveryActionListener.lookupPublishedAssessment(BeginDeliveryActionListener.java:161)
at
org.sakaiproject.tool.assessment.ui.listener.delivery.BeginDeliveryActionListener.getPublishedAssessmentBasedOnAction(BeginDeliveryActionListener.java:415)
at
org.sakaiproject.tool.assessment.ui.listener.delivery.BeginDeliveryActionListener.processAction(BeginDeliveryActionListener.java:109)
at
javax.faces.event.ActionEvent.processListener(ActionEvent.java:77)
at
javax.faces.component.UIComponentBase.broadcast(UIComponentBase.java:669)
at javax.faces.component.UICommand.broadcast(UICommand.java:317)
at javax.faces.component.UIData.broadcast(UIData.java:677)
at
javax.faces.component.UIViewRoot.broadcastEvents(UIViewRoot.java:287)
at
javax.faces.component.UIViewRoot.processApplication(UIViewRoot.java:401)
at
com.sun.faces.lifecycle.InvokeApplicationPhase.execute(InvokeApplicationPhase.java:95)
at
com.sun.faces.lifecycle.LifecycleImpl.phase(LifecycleImpl.java:245)
at
com.sun.faces.lifecycle.LifecycleImpl.execute(LifecycleImpl.java:110)
at javax.faces.webapp.FacesServlet.service(FacesServlet.java:213)
at
org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:305)
at
org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:210)
at
org.sakaiproject.util.RequestFilter.doFilter(RequestFilter.java:634)
at
org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:243)
at
org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:210)
at
org.apache.catalina.core.ApplicationDispatcher.invoke(ApplicationDispatcher.java:749)
at
org.apache.catalina.core.ApplicationDispatcher.processRequest(ApplicationDispatcher.java:487)
at
org.apache.catalina.core.ApplicationDispatcher.doForward(ApplicationDispatcher.java:412)
at
org.apache.catalina.core.ApplicationDispatcher.forward(ApplicationDispatcher.java:339)
at
org.sakaiproject.jsf.util.SamigoJsfTool.dispatch(SamigoJsfTool.java:301)
at org.sakaiproject.jsf.util.JsfTool.doPost(JsfTool.java:256)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:647)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:728)
at
org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:305)
at
org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:210)
at
org.apache.catalina.core.ApplicationDispatcher.invoke(ApplicationDispatcher.java:749)
at
org.apache.catalina.core.ApplicationDispatcher.processRequest(ApplicationDispatcher.java:487)
at
org.apache.catalina.core.ApplicationDispatcher.doForward(ApplicationDispatcher.java:379)
at
org.apache.catalina.core.ApplicationDispatcher.forward(ApplicationDispatcher.java:339)
at
org.sakaiproject.tool.impl.ActiveToolComponent$MyActiveTool.forward(ActiveToolComponent.java:511)
at
org.sakaiproject.portal.charon.SkinnableCharonPortal.forwardTool(SkinnableCharonPortal.java:1483)
at
org.sakaiproject.portal.charon.handlers.ToolHandler.doTool(ToolHandler.java:213)
at
org.sakaiproject.portal.charon.handlers.ToolHandler.doGet(ToolHandler.java:96)
at
org.sakaiproject.portal.charon.handlers.ToolHandler.doPost(ToolHandler.java:73)
at
org.sakaiproject.portal.charon.SkinnableCharonPortal.doPost(SkinnableCharonPortal.java:1273)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:647)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:728)
at
org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:305)
at
org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:210)
at
org.sakaiproject.util.RequestFilter.doFilter(RequestFilter.java:695)
at
org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:243)
at
org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:210)
at
org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:222)
at
org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:123)
at
org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:502)
at
org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:171)
at
org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:100)
at
org.apache.catalina.valves.AccessLogValve.invoke(AccessLogValve.java:953)
at
org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:118)
at
org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:408)
at
org.apache.coyote.http11.AbstractHttp11Processor.process(AbstractHttp11Processor.java:1041)
at
org.apache.coyote.AbstractProtocol$AbstractConnectionHandler.process(AbstractProtocol.java:603)
at
org.apache.tomcat.util.net.JIoEndpoint$SocketProcessor.run(JIoEndpoint.java:310)
at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:895)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:918)
at java.lang.Thread.run(Thread.java:662)
Caused by: java.lang.OutOfMemoryError: Java heap space
2013-11-01 09:41:14,635 WARN SakaiClusterService.Maintenance
org.sakaiproject.db.impl.BasicSqlService - Sql.dbRead: sql: select
SERVER_ID from SAKAI_CLUSTER where SERVER_ID = ?
opportunity-10-1383259498265
com.mysql.jdbc.exceptions.jdbc4.CommunicationsException: Communications
link failure
The last packet successfully received from the server was 6,629
milliseconds ago. The last packet sent successfully to the server was
6,629 milliseconds ago.
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native
Method)
at
sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:39)
at
sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:27)
at java.lang.reflect.Constructor.newInstance(Constructor.java:513)
at com.mysql.jdbc.Util.handleNewInstance(Util.java:411)
at
com.mysql.jdbc.SQLError.createCommunicationsException(SQLError.java:1121)
at com.mysql.jdbc.MysqlIO.reuseAndReadPacket(MysqlIO.java:3673)
at com.mysql.jdbc.MysqlIO.reuseAndReadPacket(MysqlIO.java:3562)
at com.mysql.jdbc.MysqlIO.checkErrorPacket(MysqlIO.java:4113)
at com.mysql.jdbc.MysqlIO.checkErrorPacket(MysqlIO.java:927)
at com.mysql.jdbc.MysqlIO.secureAuth411(MysqlIO.java:4689)
at com.mysql.jdbc.MysqlIO.doHandshake(MysqlIO.java:1304)
at
com.mysql.jdbc.ConnectionImpl.coreConnect(ConnectionImpl.java:2486)
at
com.mysql.jdbc.ConnectionImpl.connectOneTryOnly(ConnectionImpl.java:2519)
at
com.mysql.jdbc.ConnectionImpl.createNewIO(ConnectionImpl.java:2304)
at com.mysql.jdbc.ConnectionImpl.<init>(ConnectionImpl.java:834)
at com.mysql.jdbc.JDBC4Connection.<init>(JDBC4Connection.java:47)
at sun.reflect.GeneratedConstructorAccessor680.newInstance(Unknown
Source)
at
sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:27)
at java.lang.reflect.Constructor.newInstance(Constructor.java:513)
at com.mysql.jdbc.Util.handleNewInstance(Util.java:411)
at
com.mysql.jdbc.ConnectionImpl.getInstance(ConnectionImpl.java:416)
at
com.mysql.jdbc.NonRegisteringDriver.connect(NonRegisteringDriver.java:346)
at
org.apache.commons.dbcp.DriverConnectionFactory.createConnection(DriverConnectionFactory.java:38)
at
org.apache.commons.dbcp.PoolableConnectionFactory.makeObject(PoolableConnectionFactory.java:294)
at
org.apache.commons.pool.impl.GenericObjectPool.borrowObject(GenericObjectPool.java:1181)
at
org.apache.commons.dbcp.PoolingDataSource.getConnection(PoolingDataSource.java:96)
at
org.apache.commons.dbcp.BasicDataSource.getConnection(BasicDataSource.java:880)
at
org.sakaiproject.db.impl.BasicSqlService.borrowConnection(BasicSqlService.java:260)
at
org.sakaiproject.db.impl.BasicSqlService.dbRead(BasicSqlService.java:540)
at
org.sakaiproject.db.impl.BasicSqlService.dbRead(BasicSqlService.java:471)
at
org.sakaiproject.cluster.impl.SakaiClusterService$Maintenance.run(SakaiClusterService.java:377)
at java.lang.Thread.run(Thread.java:662)
Caused by: java.io.EOFException: Can not read response from server.
Expected to read 4 bytes, read 0 bytes before connection was unexpectedly
lost.
at com.mysql.jdbc.MysqlIO.readFully(MysqlIO.java:3119)
at com.mysql.jdbc.MysqlIO.reuseAndReadPacket(MysqlIO.java:3573)
... 26 more
2013-11-01 09:41:27,726 WARN SakaiClusterService.Maintenance
org.sakaiproject.cluster.impl.SakaiClusterService - run(): server has been
closed in cluster table, reopened: opportunity-10-1383259498265
2013-11-01 09:41:24,623 WARN http-bio-8080-exec-291
org.sakaiproject.portal.util.ErrorReporter - Bug Report bug-id:
1c74337e-3721-4fb6-812d-85f665ff1da1 user:
334d3096-e735-4597-9826-a1f6fcb44d1a usage-session:
9dbb01fb-9a47-4435-b019-3c626f9ec35c time: 01-Nov-2013 09:39:42 user
comment: null stack trace
java.lang.OutOfMemoryError: Java heap space
at java.util.Hashtable.<init>(Hashtable.java:162)
at java.util.Hashtable.<init>(Hashtable.java:183)
at
org.sakaiproject.util.BaseResourceProperties.<init>(BaseResourceProperties.java:82)
at
org.sakaiproject.util.BaseResourcePropertiesEdit.<init>(BaseResourcePropertiesEdit.java:39)
at org.sakaiproject.site.impl.BaseSite.set(BaseSite.java:575)
at org.sakaiproject.site.impl.BaseSite.<init>(BaseSite.java:234)
at
org.sakaiproject.site.impl.BaseSiteService.getCachedSite(BaseSiteService.java:599)
at
org.sakaiproject.site.impl.BaseSiteService.getDefinedSite(BaseSiteService.java:620)
at
org.sakaiproject.site.impl.BaseSiteService.getSite(BaseSiteService.java:709)
at
org.sakaiproject.site.impl.BaseSiteService.getSiteVisit(BaseSiteService.java:787)
at
org.sakaiproject.site.cover.SiteService.getSiteVisit(SiteService.java:140)
at
org.sakaiproject.portal.charon.handlers.ToolHandler.doTool(ToolHandler.java:167)
at
org.sakaiproject.portal.charon.handlers.ToolHandler.doGet(ToolHandler.java:96)
at
org.sakaiproject.portal.charon.SkinnableCharonPortal.doGet(SkinnableCharonPortal.java:894)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:621)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:728)
at
org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:305)
at
org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:210)
at org.sakaiproject.util.RequestFilter.doFilter(RequestFilter.java:695)
at
org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:243)
at
org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:210)
at
org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:222)
at
org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:123)
at
org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:502)
at
org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:171)
at
org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:100)
at
org.apache.catalina.valves.AccessLogValve.invoke(AccessLogValve.java:953)
at
org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:118)
at
org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:408)
at
org.apache.coyote.http11.AbstractHttp11Processor.process(AbstractHttp11Processor.java:1041)
at
org.apache.coyote.AbstractProtocol$AbstractConnectionHandler.process(AbstractProtocol.java:603)
at
org.apache.tomcat.util.net.JIoEndpoint$SocketProcessor.run(JIoEndpoint.java:312)
Tool Placement:
No Placement
Request:
: AuthType:null
: CharEncoding:UTF-8
: ContentLength:-1
: ContentType:null
: ContextPath:/portal
: LocalAddress:192.168.20.5
: LocalName:opportunity-10.da.org.za
: LocalPort:8080
: Method:GET
: PathInfo:/tool/00c1784d-3cb9-4b9f-ada5-4200b82376db/synMainLite
: Protocol:HTTP/1.1
: QueryString:time=1
: RemoteAddress:46.4.206.124
: RemoteHost:46.4.206.124
: RemotePort:56498
: Requested URL:
http://umthombo.da.org.za/portal/tool/00c1784d-3cb9-4b9f-ada5-4200b82376db/synMainLite
: Scheme:http
: ServerName:umthombo.da.org.za
: Headers:
: Header:accept:text/html, application/xhtml+xml, */*
: Header:referer:
http://umthombo.da.org.za/portal/tool/00c1784d-3cb9-4b9f-ada5-4200b82376db/wait
: Header:accept-language:en-ZA
: Header:user-agent:Mozilla/5.0 (compatible; MSIE 10.0; Windows NT
6.1; WOW64; Trident/6.0)
: Header:accept-encoding:gzip, deflate
: Header:dnt:1
: Header:cookie:---censored---
: Header:host:umthombo.da.org.za
: Header:cache-control:max-age=43200
: Header:connection:keep-alive
: Header:x-forwarded-for:41.13.48.159
: Parameters:
: Parameter:time:0:1
: Attributes:
:
Attribute:sakai.character.encoding.done:sakai.character.encoding.done
: Attribute:sakai.filtered:sakai.filtered
:
Attribute:sakai.session:org.sakaiproject.tool.impl.MySession at a064900
Session:
: Created:1383291288806
: LastAccess:1383291671451
: MaxInactive:3600
: Attributes:
: Attribute:portalskin:neoskin
2013-11-01 09:41:35,971 WARN http-bio-8080-exec-256
org.hibernate.util.JDBCExceptionReporter - SQL Error: 0, SQLState: null
2013-11-01 09:41:35,971 ERROR http-bio-8080-exec-256
org.hibernate.util.JDBCExceptionReporter - Already closed.
2013-11-01 09:42:09,783 WARN http-bio-8080-exec-256
org.hibernate.util.JDBCExceptionReporter - SQL Error: 0, SQLState: null
2013-11-01 09:42:09,783 ERROR http-bio-8080-exec-256
org.hibernate.util.JDBCExceptionReporter - Already closed.
2013-11-01 09:45:04,447 ERROR http-bio-8080-exec-256
org.springframework.transaction.interceptor.TransactionInterceptor -
Application exception overridden by rollback exception
java.lang.OutOfMemoryError: Java heap space
2013-11-01 09:45:04,447 ERROR http-bio-8080-exec-256
org.sakaiproject.tool.assessment.services.assessment.PublishedAssessmentService
- org.springframework.transaction.TransactionSystemException: Could not
roll back Hibernate transaction; nested exception is
org.hibernate.TransactionException: JDBC rollback failed
2013-11-01 09:45:04,447 ERROR http-bio-8080-exec-256
com.sun.faces.lifecycle.InvokeApplicationPhase -
org.springframework.transaction.TransactionSystemException: Could not roll
back Hibernate transaction; nested exception is
org.hibernate.TransactionException: JDBC rollback failed
java.lang.RuntimeException:
org.springframework.transaction.TransactionSystemException: Could not roll
back Hibernate transaction; nested exception is
org.hibernate.TransactionException: JDBC rollback failed
at
org.sakaiproject.tool.assessment.services.assessment.PublishedAssessmentService.getPublishedAssessment(PublishedAssessmentService.java:195)
at
org.sakaiproject.tool.assessment.ui.listener.delivery.BeginDeliveryActionListener.lookupPublishedAssessment(BeginDeliveryActionListener.java:161)
at
org.sakaiproject.tool.assessment.ui.listener.delivery.BeginDeliveryActionListener.getPublishedAssessmentBasedOnAction(BeginDeliveryActionListener.java:415)
at
org.sakaiproject.tool.assessment.ui.listener.delivery.BeginDeliveryActionListener.processAction(BeginDeliveryActionListener.java:109)
at
javax.faces.event.ActionEvent.processListener(ActionEvent.java:77)
at
javax.faces.component.UIComponentBase.broadcast(UIComponentBase.java:669)
at javax.faces.component.UICommand.broadcast(UICommand.java:317)
at javax.faces.component.UIData.broadcast(UIData.java:677)
at
javax.faces.component.UIViewRoot.broadcastEvents(UIViewRoot.java:287)
at
javax.faces.component.UIViewRoot.processApplication(UIViewRoot.java:401)
at
com.sun.faces.lifecycle.InvokeApplicationPhase.execute(InvokeApplicationPhase.java:95)
at
com.sun.faces.lifecycle.LifecycleImpl.phase(LifecycleImpl.java:245)
at
com.sun.faces.lifecycle.LifecycleImpl.execute(LifecycleImpl.java:110)
at javax.faces.webapp.FacesServlet.service(FacesServlet.java:213)
at
org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:305)
at
org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:210)
at
org.sakaiproject.util.RequestFilter.doFilter(RequestFilter.java:634)
at
org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:243)
at
org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:210)
at
org.apache.catalina.core.ApplicationDispatcher.invoke(ApplicationDispatcher.java:749)
at
org.apache.catalina.core.ApplicationDispatcher.processRequest(ApplicationDispatcher.java:487)
at
org.apache.catalina.core.ApplicationDispatcher.doForward(ApplicationDispatcher.java:412)
at
org.apache.catalina.core.ApplicationDispatcher.forward(ApplicationDispatcher.java:339)
at
org.sakaiproject.jsf.util.SamigoJsfTool.dispatch(SamigoJsfTool.java:301)
at org.sakaiproject.jsf.util.JsfTool.doPost(JsfTool.java:256)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:647)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:728)
at
org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:305)
at
org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:210)
at
org.apache.catalina.core.ApplicationDispatcher.invoke(ApplicationDispatcher.java:749)
at
org.apache.catalina.core.ApplicationDispatcher.processRequest(ApplicationDispatcher.java:487)
at
org.apache.catalina.core.ApplicationDispatcher.doForward(ApplicationDispatcher.java:379)
at
org.apache.catalina.core.ApplicationDispatcher.forward(ApplicationDispatcher.java:339)
at
org.sakaiproject.tool.impl.ActiveToolComponent$MyActiveTool.forward(ActiveToolComponent.java:511)
at
org.sakaiproject.portal.charon.SkinnableCharonPortal.forwardTool(SkinnableCharonPortal.java:1483)
at
org.sakaiproject.portal.charon.handlers.ToolHandler.doTool(ToolHandler.java:213)
at
org.sakaiproject.portal.charon.handlers.ToolHandler.doGet(ToolHandler.java:96)
at
org.sakaiproject.portal.charon.handlers.ToolHandler.doPost(ToolHandler.java:73)
at
org.sakaiproject.portal.charon.SkinnableCharonPortal.doPost(SkinnableCharonPortal.java:1273)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:647)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:728)
at
org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:305)
at
org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:210)
at
org.sakaiproject.util.RequestFilter.doFilter(RequestFilter.java:695)
at
org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:243)
at
org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:210)
at
org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:222)
at
org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:123)
at
org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:502)
at
org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:171)
at
org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:100)
at
org.apache.catalina.valves.AccessLogValve.invoke(AccessLogValve.java:953)
at
org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:118)
at
org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:408)
at
org.apache.coyote.http11.AbstractHttp11Processor.process(AbstractHttp11Processor.java:1041)
at
org.apache.coyote.AbstractProtocol$AbstractConnectionHandler.process(AbstractProtocol.java:603)
at
org.apache.tomcat.util.net.JIoEndpoint$SocketProcessor.run(JIoEndpoint.java:310)
at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:895)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:918)
at java.lang.Thread.run(Thread.java:662)
Caused by: org.springframework.transaction.TransactionSystemException:
Could not roll back Hibernate transaction; nested exception is
org.hibernate.TransactionException: JDBC rollback failed
at
org.springframework.orm.hibernate3.HibernateTransactionManager.doRollback(HibernateTransactionManager.java:677)
at
org.springframework.transaction.support.AbstractPlatformTransactionManager.processRollback(AbstractPlatformTransactionManager.java:823)
at
org.springframework.transaction.support.AbstractPlatformTransactionManager.rollback(AbstractPlatformTransactionManager.java:800)
at
org.springframework.transaction.interceptor.TransactionAspectSupport.completeTransactionAfterThrowing(TransactionAspectSupport.java:339)
at
org.springframework.transaction.interceptor.TransactionInterceptor.invoke(TransactionInterceptor.java:110)
at
org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:171)
at
org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:204)
at com.sun.proxy.$Proxy42.getPublishedAssessment(Unknown Source)
at
org.sakaiproject.tool.assessment.services.assessment.PublishedAssessmentService.getPublishedAssessment(PublishedAssessmentService.java:189)
... 59 more
Caused by: org.hibernate.TransactionException: JDBC rollback failed
at
org.hibernate.transaction.JDBCTransaction.rollback(JDBCTransaction.java:170)
at
org.springframework.orm.hibernate3.HibernateTransactionManager.doRollback(HibernateTransactionManager.java:674)
... 67 more
Caused by:
com.mysql.jdbc.exceptions.jdbc4.MySQLNonTransientConnectionException: No
operations allowed after connection closed.
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native
Method)
at
sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:39)
at
sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:27)
at java.lang.reflect.Constructor.newInstance(Constructor.java:513)
at com.mysql.jdbc.Util.handleNewInstance(Util.java:411)
at com.mysql.jdbc.Util.getInstance(Util.java:386)
at com.mysql.jdbc.SQLError.createSQLException(SQLError.java:1015)
at com.mysql.jdbc.SQLError.createSQLException(SQLError.java:989)
at com.mysql.jdbc.SQLError.createSQLException(SQLError.java:975)
at com.mysql.jdbc.SQLError.createSQLException(SQLError.java:920)
at
com.mysql.jdbc.ConnectionImpl.throwConnectionClosedException(ConnectionImpl.java:1304)
at
com.mysql.jdbc.ConnectionImpl.checkClosed(ConnectionImpl.java:1296)
at com.mysql.jdbc.ConnectionImpl.rollback(ConnectionImpl.java:5028)
at
org.apache.commons.dbcp.DelegatingConnection.rollback(DelegatingConnection.java:328)
at
org.apache.commons.dbcp.PoolingDataSource$PoolGuardConnectionWrapper.rollback(PoolingDataSource.java:312)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at
org.springframework.jdbc.datasource.LazyConnectionDataSourceProxy$LazyConnectionInvocationHandler.invoke(LazyConnectionDataSourceProxy.java:364)
at com.sun.proxy.$Proxy5.rollback(Unknown Source)
at
org.hibernate.transaction.JDBCTransaction.rollbackAndResetAutoCommit(JDBCTransaction.java:183)
at
org.hibernate.transaction.JDBCTransaction.rollback(JDBCTransaction.java:162)
... 68 more
Caused by: java.lang.OutOfMemoryError: Java heap space
2013-11-01 09:45:29,599 ERROR http-bio-8080-exec-295
edu.amc.sakai.user.JLDAPDirectoryProvider - getUser() failed [eid:
7306145161083]
LDAPException: Client request timed out (85) LDAP Timeout
at com.novell.ldap.Message$Timeout.run(Unknown Source)
2013-11-01 09:45:29,600 ERROR http-bio-8080-exec-289
edu.amc.sakai.user.JLDAPDirectoryProvider - getUser() failed [eid:
5804290142082]
LDAPException: Client request timed out (85) LDAP Timeout
at com.novell.ldap.Message$Timeout.run(Unknown Source)
2013-11-01 09:45:29,600 ERROR http-bio-8080-exec-284
edu.amc.sakai.user.JLDAPDirectoryProvider - getUser() failed [eid:
5911295089083]
LDAPException: Client request timed out (85) LDAP Timeout
at com.novell.ldap.Message$Timeout.run(Unknown Source)
2013-11-01 09:45:34,406 WARN http-bio-8080-exec-302
org.sakaiproject.portal.util.ErrorReporter - Bug Report bug-id:
4b8c41bb-ec01-41ed-8186-02448c497c6c user: null usage-session: null time:
01-Nov-2013 09:45:29 user comment: null stack trace
org.sakaiproject.portal.api.PortalHandlerException:
java.lang.RuntimeException: authenticateUser(): LDAPException during
authentication attempt [userLogin = support][result code = LDAP
Timeout][error message = null]
at
org.sakaiproject.portal.charon.handlers.XLoginHandler.doPost(XLoginHandler.java:50)
caused by: java.lang.RuntimeException: authenticateUser(): LDAPException
during authentication attempt [userLogin = support][result code = LDAP
Timeout][error message = null]
at
org.sakaiproject.user.impl.BaseUserDirectoryService.getProviderAuthenticatedUser(BaseUserDirectoryService.java:1597)
caused by: LDAPException: Client request timed out (85) LDAP Timeout
at com.novell.ldap.Message$Timeout.run(null:-1)
Tool Placement:
No Placement
Request:
: AuthType:null
: CharEncoding:UTF-8
: ContentLength:33
: ContentType:application/x-www-form-urlencoded
: ContextPath:/portal
: LocalAddress:192.168.20.5
: LocalName:opportunity-10.da.org.za
: LocalPort:8080
: Method:POST
: PathInfo:/xlogin
: Protocol:HTTP/1.1
: QueryString:null
: RemoteAddress:46.4.206.124
: RemoteHost:46.4.206.124
: RemotePort:56684
: Requested URL:http://umthombo.da.org.za/portal/xlogin
: Scheme:http
: ServerName:umthombo.da.org.za
: Headers:
: Header:host:umthombo.da.org.za
: Header:content-length:33
:
Header:accept:text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,*/*;q=0.8
: Header:origin:http://umthombo.da.org.za
: Header:user-agent:Mozilla/5.0 (Windows NT 6.2; WOW64)
AppleWebKit/537.36 (KHTML, like Gecko) Chrome/30.0.1599.101 Safari/537.36
: Header:content-type:application/x-www-form-urlencoded
: Header:referer:http://umthombo.da.org.za/portal
: Header:accept-encoding:gzip,deflate,sdch
: Header:accept-language:en-US,en;q=0.8
: Header:cookie:---censored---
: Header:via:1.1 proxy.iuncapped.co.za (squid/3.1.10)
: Header:x-forwarded-for:192.168.35.1
: Header:x-forwarded-for:41.216.135.90
: Header:cache-control:max-age=259200
: Header:connection:keep-alive
: Parameters:
: Parameter:eid:0:----censored----
: Parameter:pw:0:----censored----
: Parameter:submit:0:Login
: Attributes:
:
Attribute:sakai.character.encoding.done:sakai.character.encoding.done
: Attribute:sakai.filtered:sakai.filtered
:
Attribute:sakai.session:org.sakaiproject.tool.impl.MySession at 20cce106
Session:
: Created:1383290355854
: LastAccess:1383291929601
: MaxInactive:3600
: Attributes:
: Attribute:portalskin:neoskin
2013-11-01 09:45:34,406 WARN http-bio-8080-exec-302
org.sakaiproject.email.impl.BasicEmailService - send:
javax.mail.internet.AddressException: Illegal address in string ``''
Exception in thread "Thread-12" java.lang.OutOfMemoryError: Java heap space
-- Warwick Bruce Chapman | +27 83 7797 094 | http://warwickchapman.com
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://collab.sakaiproject.org/pipermail/sakai-dev/attachments/20131101/8b70a2aa/attachment.html
More information about the sakai-dev
mailing list