[Building Sakai] Application Servers Crashing
Warwick Chapman
warwickchapman at gmail.com
Fri Nov 1 01:27:13 PDT 2013
The three are running at the moment, with 63 sessions shared between them:
root at opportunity:~# uname -a
Linux opportunity 2.6.32-19-pve #1 SMP Wed May 15 07:32:52 CEST 2013 x86_64
x86_64 x86_64 GNU/Linux
root at opportunity:~# free -m
total used free shared buffers cached
Mem: 4096 3679 416 0 0 1968
-/+ buffers/cache: 1711 2384
Swap: 4096 6 4089
root at opportunity:~# w
10:23:19 up 1 day, 14:23, 1 user, load average: 0,16, 0,08, 0,04
---
root at opportunity-01:~# uname -a
Linux opportunity-01 2.6.32-19-pve #1 SMP Wed May 15 07:32:52 CEST 2013
x86_64 x86_64 x86_64 GNU/Linux
root at opportunity-01:~# free -m
total used free shared buffers cached
Mem: 4096 2662 1433 0 0 954
-/+ buffers/cache: 1708 2387
Swap: 4096 3 4092
root at opportunity-01:~# w
10:23:54 up 22:56, 1 user, load average: 0,03, 0,05, 0,00
---
root at opportunity-10:~# uname -a
Linux opportunity-10 2.6.32-20-pve #1 SMP Wed May 15 08:23:27 CEST 2013
x86_64 x86_64 x86_64 GNU/Linux
root at opportunity-10:~# free -m
total used free shared buffers cached
Mem: 4096 1778 2317 0 0 146
-/+ buffers/cache: 1632 2463
Swap: 4096 0 4096
root at opportunity-10:~# w
10:24:40 up 10:40, 2 users, load average: 0,00, 0,02, 0,13
-- Warwick Bruce Chapman | +27 83 7797 094 | http://warwickchapman.com
On Fri, Nov 1, 2013 at 10:04 AM, Warwick Chapman
<warwickchapman at gmail.com>wrote:
> Hi All
>
> I'm going mad here. Surely we can expect a four processor, 4G appserver
> to handle more than 50 sessions without crashing?
>
> Mine keep crashing and its causing havoc.
>
> As they approach 40 sessions, the system load which is usually below 0.10
> start climbing, and then hits 1.00 and eventually I get the
> "java.lang.OutOfMemoryError: Java heap space" error.
>
> *Details*
>
> 1. Ubuntu 12.04 OpenVZ on Proxmox with 4G RAM and 4 processors allocated.
>
> 2. Oracle Java 6:
> root at opportunity-10:~# java -version
> java version "1.6.0_45"
> Java(TM) SE Runtime Environment (build 1.6.0_45-b06)
> Java HotSpot(TM) 64-Bit Server VM (build 20.45-b01, mixed mode)
>
> 3. Environment variables:
> root at opportunity-10:~# set | grep JAVA
> JAVA_HOME=/usr/lib/jvm/java-6-oracle
> JAVA_OPTS='-server -Xms512m -Xmx1024m -XX:PermSize=128m
> -XX:MaxPermSize=512m -XX:NewSize=192m -XX:MaxNewSize=384m
> -Djava.awt.headless=true -Dhttp.agent=Sakai
> -Dorg.apache.jasper.compiler.Parser.STRICT_QUOTE_ESCAPING=false
> -Dsun.lang.ClassLoader.allowArraySyntax=true -Duser.language=en
> -Duser.region=ZA -Dfile.encoding=UTF-8'
>
> 4. Sakai 2.9.3 built independently on each app server (also tried building
> on one and deploying to others). Custom providers, SMS and only patch is
> for UCT SMS.
>
> 5. Process:
> 31032 pts/2 Sl 1:44 /usr/lib/jvm/java-6-oracle/bin/java
> -Djava.util.logging.config.file=/opt/tomcat/conf/logging.properties
> -Djava.util.logging.manager=org.apache.juli.ClassLoaderLogManager -server
> -Xms512m -Xmx1024m -XX:PermSize=128m -XX:MaxPermSize=512m -XX:NewSize=192m
> -XX:MaxNewSize=384m -Djava.awt.headless=true -Dhttp.agent=Sakai
> -Dorg.apache.jasper.compiler.Parser.STRICT_QUOTE_ESCAPING=false
> -Dsun.lang.ClassLoader.allowArraySyntax=true -Duser.language=en
> -Duser.region=ZA -Dfile.encoding=UTF-8
> -Djava.endorsed.dirs=/opt/tomcat/endorsed -classpath
> /opt/tomcat/bin/bootstrap.jar:/opt/tomcat/bin/tomcat-juli.jar
> -Dcatalina.base=/opt/tomcat -Dcatalina.home=/opt/tomcat
> -Djava.io.tmpdir=/opt/tomcat/temp org.apache.catalina.startup.Bootstrap
> start
>
> 6. Database server and LDAP server are quiet and not battling with load at
> all.
>
> 7. The underlying host machines are quiet and load is not affecting them
> at all and memory usage is safely below available.
>
> See catalina for the most recent crash:
> 2013-11-01 09:37:18,497 WARN http-bio-8080-exec-281
> org.sakaiproject.portal.charon.site.DefaultSiteViewImpl - Failed to set
> canAddSite for current user. Defaulting to false ...
> Exception in thread "PageSavingThread-sakai.profile2"
> java.lang.OutOfMemoryError: Java heap space
> 2013-11-01 09:41:04,553 ERROR http-bio-8080-exec-256
> org.hibernate.transaction.JDBCTransaction - JDBC rollback failed
> com.mysql.jdbc.exceptions.jdbc4.MySQLNonTransientConnectionException: No
> operations allowed after connection closed.
> at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native
> Method)
> at
> sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:39)
> at
> sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:27)
> at java.lang.reflect.Constructor.newInstance(Constructor.java:513)
> at com.mysql.jdbc.Util.handleNewInstance(Util.java:411)
> at com.mysql.jdbc.Util.getInstance(Util.java:386)
> at com.mysql.jdbc.SQLError.createSQLException(SQLError.java:1015)
> at com.mysql.jdbc.SQLError.createSQLException(SQLError.java:989)
> at com.mysql.jdbc.SQLError.createSQLException(SQLError.java:975)
> at com.mysql.jdbc.SQLError.createSQLException(SQLError.java:920)
> at
> com.mysql.jdbc.ConnectionImpl.throwConnectionClosedException(ConnectionImpl.java:1304)
> at
> com.mysql.jdbc.ConnectionImpl.checkClosed(ConnectionImpl.java:1296)
> at com.mysql.jdbc.ConnectionImpl.rollback(ConnectionImpl.java:5028)
> at
> org.apache.commons.dbcp.DelegatingConnection.rollback(DelegatingConnection.java:328)
> at
> org.apache.commons.dbcp.PoolingDataSource$PoolGuardConnectionWrapper.rollback(PoolingDataSource.java:312)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:597)
> at
> org.springframework.jdbc.datasource.LazyConnectionDataSourceProxy$LazyConnectionInvocationHandler.invoke(LazyConnectionDataSourceProxy.java:364)
> at com.sun.proxy.$Proxy5.rollback(Unknown Source)
> at
> org.hibernate.transaction.JDBCTransaction.rollbackAndResetAutoCommit(JDBCTransaction.java:183)
> at
> org.hibernate.transaction.JDBCTransaction.rollback(JDBCTransaction.java:162)
> at
> org.springframework.orm.hibernate3.HibernateTransactionManager.doRollback(HibernateTransactionManager.java:674)
> at
> org.springframework.transaction.support.AbstractPlatformTransactionManager.processRollback(AbstractPlatformTransactionManager.java:823)
> at
> org.springframework.transaction.support.AbstractPlatformTransactionManager.rollback(AbstractPlatformTransactionManager.java:800)
> at
> org.springframework.transaction.interceptor.TransactionAspectSupport.completeTransactionAfterThrowing(TransactionAspectSupport.java:339)
> at
> org.springframework.transaction.interceptor.TransactionInterceptor.invoke(TransactionInterceptor.java:110)
> at
> org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:171)
> at
> org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:204)
> at com.sun.proxy.$Proxy42.getPublishedAssessment(Unknown Source)
> at
> org.sakaiproject.tool.assessment.services.assessment.PublishedAssessmentService.getPublishedAssessment(PublishedAssessmentService.java:189)
> at
> org.sakaiproject.tool.assessment.ui.listener.delivery.BeginDeliveryActionListener.lookupPublishedAssessment(BeginDeliveryActionListener.java:161)
> at
> org.sakaiproject.tool.assessment.ui.listener.delivery.BeginDeliveryActionListener.getPublishedAssessmentBasedOnAction(BeginDeliveryActionListener.java:415)
> at
> org.sakaiproject.tool.assessment.ui.listener.delivery.BeginDeliveryActionListener.processAction(BeginDeliveryActionListener.java:109)
> at
> javax.faces.event.ActionEvent.processListener(ActionEvent.java:77)
> at
> javax.faces.component.UIComponentBase.broadcast(UIComponentBase.java:669)
> at javax.faces.component.UICommand.broadcast(UICommand.java:317)
> at javax.faces.component.UIData.broadcast(UIData.java:677)
> at
> javax.faces.component.UIViewRoot.broadcastEvents(UIViewRoot.java:287)
> at
> javax.faces.component.UIViewRoot.processApplication(UIViewRoot.java:401)
> at
> com.sun.faces.lifecycle.InvokeApplicationPhase.execute(InvokeApplicationPhase.java:95)
> at
> com.sun.faces.lifecycle.LifecycleImpl.phase(LifecycleImpl.java:245)
> at
> com.sun.faces.lifecycle.LifecycleImpl.execute(LifecycleImpl.java:110)
> at javax.faces.webapp.FacesServlet.service(FacesServlet.java:213)
> at
> org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:305)
> at
> org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:210)
> at
> org.sakaiproject.util.RequestFilter.doFilter(RequestFilter.java:634)
> at
> org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:243)
> at
> org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:210)
> at
> org.apache.catalina.core.ApplicationDispatcher.invoke(ApplicationDispatcher.java:749)
> at
> org.apache.catalina.core.ApplicationDispatcher.processRequest(ApplicationDispatcher.java:487)
> at
> org.apache.catalina.core.ApplicationDispatcher.doForward(ApplicationDispatcher.java:412)
> at
> org.apache.catalina.core.ApplicationDispatcher.forward(ApplicationDispatcher.java:339)
> at
> org.sakaiproject.jsf.util.SamigoJsfTool.dispatch(SamigoJsfTool.java:301)
> at org.sakaiproject.jsf.util.JsfTool.doPost(JsfTool.java:256)
> at javax.servlet.http.HttpServlet.service(HttpServlet.java:647)
> at javax.servlet.http.HttpServlet.service(HttpServlet.java:728)
> at
> org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:305)
> at
> org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:210)
> at
> org.apache.catalina.core.ApplicationDispatcher.invoke(ApplicationDispatcher.java:749)
> at
> org.apache.catalina.core.ApplicationDispatcher.processRequest(ApplicationDispatcher.java:487)
> at
> org.apache.catalina.core.ApplicationDispatcher.doForward(ApplicationDispatcher.java:379)
> at
> org.apache.catalina.core.ApplicationDispatcher.forward(ApplicationDispatcher.java:339)
> at
> org.sakaiproject.tool.impl.ActiveToolComponent$MyActiveTool.forward(ActiveToolComponent.java:511)
> at
> org.sakaiproject.portal.charon.SkinnableCharonPortal.forwardTool(SkinnableCharonPortal.java:1483)
> at
> org.sakaiproject.portal.charon.handlers.ToolHandler.doTool(ToolHandler.java:213)
> at
> org.sakaiproject.portal.charon.handlers.ToolHandler.doGet(ToolHandler.java:96)
> at
> org.sakaiproject.portal.charon.handlers.ToolHandler.doPost(ToolHandler.java:73)
> at
> org.sakaiproject.portal.charon.SkinnableCharonPortal.doPost(SkinnableCharonPortal.java:1273)
> at javax.servlet.http.HttpServlet.service(HttpServlet.java:647)
> at javax.servlet.http.HttpServlet.service(HttpServlet.java:728)
> at
> org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:305)
> at
> org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:210)
> at
> org.sakaiproject.util.RequestFilter.doFilter(RequestFilter.java:695)
> at
> org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:243)
> at
> org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:210)
> at
> org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:222)
> at
> org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:123)
> at
> org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:502)
> at
> org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:171)
> at
> org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:100)
> at
> org.apache.catalina.valves.AccessLogValve.invoke(AccessLogValve.java:953)
> at
> org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:118)
> at
> org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:408)
> at
> org.apache.coyote.http11.AbstractHttp11Processor.process(AbstractHttp11Processor.java:1041)
> at
> org.apache.coyote.AbstractProtocol$AbstractConnectionHandler.process(AbstractProtocol.java:603)
> at
> org.apache.tomcat.util.net.JIoEndpoint$SocketProcessor.run(JIoEndpoint.java:310)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:895)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:918)
> at java.lang.Thread.run(Thread.java:662)
> Caused by: java.lang.OutOfMemoryError: Java heap space
> 2013-11-01 09:41:14,635 WARN SakaiClusterService.Maintenance
> org.sakaiproject.db.impl.BasicSqlService - Sql.dbRead: sql: select
> SERVER_ID from SAKAI_CLUSTER where SERVER_ID = ?
> opportunity-10-1383259498265
> com.mysql.jdbc.exceptions.jdbc4.CommunicationsException: Communications
> link failure
>
> The last packet successfully received from the server was 6,629
> milliseconds ago. The last packet sent successfully to the server was
> 6,629 milliseconds ago.
> at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native
> Method)
> at
> sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:39)
> at
> sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:27)
> at java.lang.reflect.Constructor.newInstance(Constructor.java:513)
> at com.mysql.jdbc.Util.handleNewInstance(Util.java:411)
> at
> com.mysql.jdbc.SQLError.createCommunicationsException(SQLError.java:1121)
> at com.mysql.jdbc.MysqlIO.reuseAndReadPacket(MysqlIO.java:3673)
> at com.mysql.jdbc.MysqlIO.reuseAndReadPacket(MysqlIO.java:3562)
> at com.mysql.jdbc.MysqlIO.checkErrorPacket(MysqlIO.java:4113)
> at com.mysql.jdbc.MysqlIO.checkErrorPacket(MysqlIO.java:927)
> at com.mysql.jdbc.MysqlIO.secureAuth411(MysqlIO.java:4689)
> at com.mysql.jdbc.MysqlIO.doHandshake(MysqlIO.java:1304)
> at
> com.mysql.jdbc.ConnectionImpl.coreConnect(ConnectionImpl.java:2486)
> at
> com.mysql.jdbc.ConnectionImpl.connectOneTryOnly(ConnectionImpl.java:2519)
> at
> com.mysql.jdbc.ConnectionImpl.createNewIO(ConnectionImpl.java:2304)
> at com.mysql.jdbc.ConnectionImpl.<init>(ConnectionImpl.java:834)
> at com.mysql.jdbc.JDBC4Connection.<init>(JDBC4Connection.java:47)
> at sun.reflect.GeneratedConstructorAccessor680.newInstance(Unknown
> Source)
> at
> sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:27)
> at java.lang.reflect.Constructor.newInstance(Constructor.java:513)
> at com.mysql.jdbc.Util.handleNewInstance(Util.java:411)
> at
> com.mysql.jdbc.ConnectionImpl.getInstance(ConnectionImpl.java:416)
> at
> com.mysql.jdbc.NonRegisteringDriver.connect(NonRegisteringDriver.java:346)
> at
> org.apache.commons.dbcp.DriverConnectionFactory.createConnection(DriverConnectionFactory.java:38)
> at
> org.apache.commons.dbcp.PoolableConnectionFactory.makeObject(PoolableConnectionFactory.java:294)
> at
> org.apache.commons.pool.impl.GenericObjectPool.borrowObject(GenericObjectPool.java:1181)
> at
> org.apache.commons.dbcp.PoolingDataSource.getConnection(PoolingDataSource.java:96)
> at
> org.apache.commons.dbcp.BasicDataSource.getConnection(BasicDataSource.java:880)
> at
> org.sakaiproject.db.impl.BasicSqlService.borrowConnection(BasicSqlService.java:260)
> at
> org.sakaiproject.db.impl.BasicSqlService.dbRead(BasicSqlService.java:540)
> at
> org.sakaiproject.db.impl.BasicSqlService.dbRead(BasicSqlService.java:471)
> at
> org.sakaiproject.cluster.impl.SakaiClusterService$Maintenance.run(SakaiClusterService.java:377)
> at java.lang.Thread.run(Thread.java:662)
> Caused by: java.io.EOFException: Can not read response from server.
> Expected to read 4 bytes, read 0 bytes before connection was unexpectedly
> lost.
> at com.mysql.jdbc.MysqlIO.readFully(MysqlIO.java:3119)
> at com.mysql.jdbc.MysqlIO.reuseAndReadPacket(MysqlIO.java:3573)
> ... 26 more
> 2013-11-01 09:41:27,726 WARN SakaiClusterService.Maintenance
> org.sakaiproject.cluster.impl.SakaiClusterService - run(): server has been
> closed in cluster table, reopened: opportunity-10-1383259498265
> 2013-11-01 09:41:24,623 WARN http-bio-8080-exec-291
> org.sakaiproject.portal.util.ErrorReporter - Bug Report bug-id:
> 1c74337e-3721-4fb6-812d-85f665ff1da1 user:
> 334d3096-e735-4597-9826-a1f6fcb44d1a usage-session:
> 9dbb01fb-9a47-4435-b019-3c626f9ec35c time: 01-Nov-2013 09:39:42 user
> comment: null stack trace
> java.lang.OutOfMemoryError: Java heap space
> at java.util.Hashtable.<init>(Hashtable.java:162)
> at java.util.Hashtable.<init>(Hashtable.java:183)
> at
> org.sakaiproject.util.BaseResourceProperties.<init>(BaseResourceProperties.java:82)
> at
> org.sakaiproject.util.BaseResourcePropertiesEdit.<init>(BaseResourcePropertiesEdit.java:39)
> at org.sakaiproject.site.impl.BaseSite.set(BaseSite.java:575)
> at org.sakaiproject.site.impl.BaseSite.<init>(BaseSite.java:234)
> at
> org.sakaiproject.site.impl.BaseSiteService.getCachedSite(BaseSiteService.java:599)
> at
> org.sakaiproject.site.impl.BaseSiteService.getDefinedSite(BaseSiteService.java:620)
> at
> org.sakaiproject.site.impl.BaseSiteService.getSite(BaseSiteService.java:709)
> at
> org.sakaiproject.site.impl.BaseSiteService.getSiteVisit(BaseSiteService.java:787)
> at
> org.sakaiproject.site.cover.SiteService.getSiteVisit(SiteService.java:140)
> at
> org.sakaiproject.portal.charon.handlers.ToolHandler.doTool(ToolHandler.java:167)
> at
> org.sakaiproject.portal.charon.handlers.ToolHandler.doGet(ToolHandler.java:96)
> at
> org.sakaiproject.portal.charon.SkinnableCharonPortal.doGet(SkinnableCharonPortal.java:894)
> at javax.servlet.http.HttpServlet.service(HttpServlet.java:621)
> at javax.servlet.http.HttpServlet.service(HttpServlet.java:728)
> at
> org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:305)
> at
> org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:210)
> at org.sakaiproject.util.RequestFilter.doFilter(RequestFilter.java:695)
> at
> org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:243)
> at
> org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:210)
> at
> org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:222)
> at
> org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:123)
> at
> org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:502)
> at
> org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:171)
> at
> org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:100)
> at
> org.apache.catalina.valves.AccessLogValve.invoke(AccessLogValve.java:953)
> at
> org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:118)
> at
> org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:408)
> at
> org.apache.coyote.http11.AbstractHttp11Processor.process(AbstractHttp11Processor.java:1041)
> at
> org.apache.coyote.AbstractProtocol$AbstractConnectionHandler.process(AbstractProtocol.java:603)
> at
> org.apache.tomcat.util.net.JIoEndpoint$SocketProcessor.run(JIoEndpoint.java:312)
>
> Tool Placement:
> No Placement
>
> Request:
> : AuthType:null
> : CharEncoding:UTF-8
> : ContentLength:-1
> : ContentType:null
> : ContextPath:/portal
> : LocalAddress:192.168.20.5
> : LocalName:opportunity-10.da.org.za
> : LocalPort:8080
> : Method:GET
> : PathInfo:/tool/00c1784d-3cb9-4b9f-ada5-4200b82376db/synMainLite
> : Protocol:HTTP/1.1
> : QueryString:time=1
> : RemoteAddress:46.4.206.124
> : RemoteHost:46.4.206.124
> : RemotePort:56498
> : Requested URL:
> http://umthombo.da.org.za/portal/tool/00c1784d-3cb9-4b9f-ada5-4200b82376db/synMainLite
> : Scheme:http
> : ServerName:umthombo.da.org.za
> : Headers:
> : Header:accept:text/html, application/xhtml+xml, */*
> : Header:referer:
> http://umthombo.da.org.za/portal/tool/00c1784d-3cb9-4b9f-ada5-4200b82376db/wait
> : Header:accept-language:en-ZA
> : Header:user-agent:Mozilla/5.0 (compatible; MSIE 10.0; Windows NT
> 6.1; WOW64; Trident/6.0)
> : Header:accept-encoding:gzip, deflate
> : Header:dnt:1
> : Header:cookie:---censored---
> : Header:host:umthombo.da.org.za
> : Header:cache-control:max-age=43200
> : Header:connection:keep-alive
> : Header:x-forwarded-for:41.13.48.159
> : Parameters:
> : Parameter:time:0:1
> : Attributes:
> :
> Attribute:sakai.character.encoding.done:sakai.character.encoding.done
> : Attribute:sakai.filtered:sakai.filtered
> :
> Attribute:sakai.session:org.sakaiproject.tool.impl.MySession at a064900
> Session:
> : Created:1383291288806
> : LastAccess:1383291671451
> : MaxInactive:3600
> : Attributes:
> : Attribute:portalskin:neoskin
>
> 2013-11-01 09:41:35,971 WARN http-bio-8080-exec-256
> org.hibernate.util.JDBCExceptionReporter - SQL Error: 0, SQLState: null
> 2013-11-01 09:41:35,971 ERROR http-bio-8080-exec-256
> org.hibernate.util.JDBCExceptionReporter - Already closed.
> 2013-11-01 09:42:09,783 WARN http-bio-8080-exec-256
> org.hibernate.util.JDBCExceptionReporter - SQL Error: 0, SQLState: null
> 2013-11-01 09:42:09,783 ERROR http-bio-8080-exec-256
> org.hibernate.util.JDBCExceptionReporter - Already closed.
> 2013-11-01 09:45:04,447 ERROR http-bio-8080-exec-256
> org.springframework.transaction.interceptor.TransactionInterceptor -
> Application exception overridden by rollback exception
> java.lang.OutOfMemoryError: Java heap space
> 2013-11-01 09:45:04,447 ERROR http-bio-8080-exec-256
> org.sakaiproject.tool.assessment.services.assessment.PublishedAssessmentService
> - org.springframework.transaction.TransactionSystemException: Could not
> roll back Hibernate transaction; nested exception is
> org.hibernate.TransactionException: JDBC rollback failed
> 2013-11-01 09:45:04,447 ERROR http-bio-8080-exec-256
> com.sun.faces.lifecycle.InvokeApplicationPhase -
> org.springframework.transaction.TransactionSystemException: Could not roll
> back Hibernate transaction; nested exception is
> org.hibernate.TransactionException: JDBC rollback failed
> java.lang.RuntimeException:
> org.springframework.transaction.TransactionSystemException: Could not roll
> back Hibernate transaction; nested exception is
> org.hibernate.TransactionException: JDBC rollback failed
> at
> org.sakaiproject.tool.assessment.services.assessment.PublishedAssessmentService.getPublishedAssessment(PublishedAssessmentService.java:195)
> at
> org.sakaiproject.tool.assessment.ui.listener.delivery.BeginDeliveryActionListener.lookupPublishedAssessment(BeginDeliveryActionListener.java:161)
> at
> org.sakaiproject.tool.assessment.ui.listener.delivery.BeginDeliveryActionListener.getPublishedAssessmentBasedOnAction(BeginDeliveryActionListener.java:415)
> at
> org.sakaiproject.tool.assessment.ui.listener.delivery.BeginDeliveryActionListener.processAction(BeginDeliveryActionListener.java:109)
> at
> javax.faces.event.ActionEvent.processListener(ActionEvent.java:77)
> at
> javax.faces.component.UIComponentBase.broadcast(UIComponentBase.java:669)
> at javax.faces.component.UICommand.broadcast(UICommand.java:317)
> at javax.faces.component.UIData.broadcast(UIData.java:677)
> at
> javax.faces.component.UIViewRoot.broadcastEvents(UIViewRoot.java:287)
> at
> javax.faces.component.UIViewRoot.processApplication(UIViewRoot.java:401)
> at
> com.sun.faces.lifecycle.InvokeApplicationPhase.execute(InvokeApplicationPhase.java:95)
> at
> com.sun.faces.lifecycle.LifecycleImpl.phase(LifecycleImpl.java:245)
> at
> com.sun.faces.lifecycle.LifecycleImpl.execute(LifecycleImpl.java:110)
> at javax.faces.webapp.FacesServlet.service(FacesServlet.java:213)
> at
> org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:305)
> at
> org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:210)
> at
> org.sakaiproject.util.RequestFilter.doFilter(RequestFilter.java:634)
> at
> org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:243)
> at
> org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:210)
> at
> org.apache.catalina.core.ApplicationDispatcher.invoke(ApplicationDispatcher.java:749)
> at
> org.apache.catalina.core.ApplicationDispatcher.processRequest(ApplicationDispatcher.java:487)
> at
> org.apache.catalina.core.ApplicationDispatcher.doForward(ApplicationDispatcher.java:412)
> at
> org.apache.catalina.core.ApplicationDispatcher.forward(ApplicationDispatcher.java:339)
> at
> org.sakaiproject.jsf.util.SamigoJsfTool.dispatch(SamigoJsfTool.java:301)
> at org.sakaiproject.jsf.util.JsfTool.doPost(JsfTool.java:256)
> at javax.servlet.http.HttpServlet.service(HttpServlet.java:647)
> at javax.servlet.http.HttpServlet.service(HttpServlet.java:728)
> at
> org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:305)
> at
> org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:210)
> at
> org.apache.catalina.core.ApplicationDispatcher.invoke(ApplicationDispatcher.java:749)
> at
> org.apache.catalina.core.ApplicationDispatcher.processRequest(ApplicationDispatcher.java:487)
> at
> org.apache.catalina.core.ApplicationDispatcher.doForward(ApplicationDispatcher.java:379)
> at
> org.apache.catalina.core.ApplicationDispatcher.forward(ApplicationDispatcher.java:339)
> at
> org.sakaiproject.tool.impl.ActiveToolComponent$MyActiveTool.forward(ActiveToolComponent.java:511)
> at
> org.sakaiproject.portal.charon.SkinnableCharonPortal.forwardTool(SkinnableCharonPortal.java:1483)
> at
> org.sakaiproject.portal.charon.handlers.ToolHandler.doTool(ToolHandler.java:213)
> at
> org.sakaiproject.portal.charon.handlers.ToolHandler.doGet(ToolHandler.java:96)
> at
> org.sakaiproject.portal.charon.handlers.ToolHandler.doPost(ToolHandler.java:73)
> at
> org.sakaiproject.portal.charon.SkinnableCharonPortal.doPost(SkinnableCharonPortal.java:1273)
> at javax.servlet.http.HttpServlet.service(HttpServlet.java:647)
> at javax.servlet.http.HttpServlet.service(HttpServlet.java:728)
> at
> org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:305)
> at
> org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:210)
> at
> org.sakaiproject.util.RequestFilter.doFilter(RequestFilter.java:695)
> at
> org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:243)
> at
> org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:210)
> at
> org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:222)
> at
> org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:123)
> at
> org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:502)
> at
> org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:171)
> at
> org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:100)
> at
> org.apache.catalina.valves.AccessLogValve.invoke(AccessLogValve.java:953)
> at
> org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:118)
> at
> org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:408)
> at
> org.apache.coyote.http11.AbstractHttp11Processor.process(AbstractHttp11Processor.java:1041)
> at
> org.apache.coyote.AbstractProtocol$AbstractConnectionHandler.process(AbstractProtocol.java:603)
> at
> org.apache.tomcat.util.net.JIoEndpoint$SocketProcessor.run(JIoEndpoint.java:310)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:895)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:918)
> at java.lang.Thread.run(Thread.java:662)
> Caused by: org.springframework.transaction.TransactionSystemException:
> Could not roll back Hibernate transaction; nested exception is
> org.hibernate.TransactionException: JDBC rollback failed
> at
> org.springframework.orm.hibernate3.HibernateTransactionManager.doRollback(HibernateTransactionManager.java:677)
> at
> org.springframework.transaction.support.AbstractPlatformTransactionManager.processRollback(AbstractPlatformTransactionManager.java:823)
> at
> org.springframework.transaction.support.AbstractPlatformTransactionManager.rollback(AbstractPlatformTransactionManager.java:800)
> at
> org.springframework.transaction.interceptor.TransactionAspectSupport.completeTransactionAfterThrowing(TransactionAspectSupport.java:339)
> at
> org.springframework.transaction.interceptor.TransactionInterceptor.invoke(TransactionInterceptor.java:110)
> at
> org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:171)
> at
> org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:204)
> at com.sun.proxy.$Proxy42.getPublishedAssessment(Unknown Source)
> at
> org.sakaiproject.tool.assessment.services.assessment.PublishedAssessmentService.getPublishedAssessment(PublishedAssessmentService.java:189)
> ... 59 more
> Caused by: org.hibernate.TransactionException: JDBC rollback failed
> at
> org.hibernate.transaction.JDBCTransaction.rollback(JDBCTransaction.java:170)
> at
> org.springframework.orm.hibernate3.HibernateTransactionManager.doRollback(HibernateTransactionManager.java:674)
> ... 67 more
> Caused by:
> com.mysql.jdbc.exceptions.jdbc4.MySQLNonTransientConnectionException: No
> operations allowed after connection closed.
> at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native
> Method)
> at
> sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:39)
> at
> sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:27)
> at java.lang.reflect.Constructor.newInstance(Constructor.java:513)
> at com.mysql.jdbc.Util.handleNewInstance(Util.java:411)
> at com.mysql.jdbc.Util.getInstance(Util.java:386)
> at com.mysql.jdbc.SQLError.createSQLException(SQLError.java:1015)
> at com.mysql.jdbc.SQLError.createSQLException(SQLError.java:989)
> at com.mysql.jdbc.SQLError.createSQLException(SQLError.java:975)
> at com.mysql.jdbc.SQLError.createSQLException(SQLError.java:920)
> at
> com.mysql.jdbc.ConnectionImpl.throwConnectionClosedException(ConnectionImpl.java:1304)
> at
> com.mysql.jdbc.ConnectionImpl.checkClosed(ConnectionImpl.java:1296)
> at com.mysql.jdbc.ConnectionImpl.rollback(ConnectionImpl.java:5028)
> at
> org.apache.commons.dbcp.DelegatingConnection.rollback(DelegatingConnection.java:328)
> at
> org.apache.commons.dbcp.PoolingDataSource$PoolGuardConnectionWrapper.rollback(PoolingDataSource.java:312)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:597)
> at
> org.springframework.jdbc.datasource.LazyConnectionDataSourceProxy$LazyConnectionInvocationHandler.invoke(LazyConnectionDataSourceProxy.java:364)
> at com.sun.proxy.$Proxy5.rollback(Unknown Source)
> at
> org.hibernate.transaction.JDBCTransaction.rollbackAndResetAutoCommit(JDBCTransaction.java:183)
> at
> org.hibernate.transaction.JDBCTransaction.rollback(JDBCTransaction.java:162)
> ... 68 more
> Caused by: java.lang.OutOfMemoryError: Java heap space
> 2013-11-01 09:45:29,599 ERROR http-bio-8080-exec-295
> edu.amc.sakai.user.JLDAPDirectoryProvider - getUser() failed [eid:
> 7306145161083]
> LDAPException: Client request timed out (85) LDAP Timeout
> at com.novell.ldap.Message$Timeout.run(Unknown Source)
> 2013-11-01 09:45:29,600 ERROR http-bio-8080-exec-289
> edu.amc.sakai.user.JLDAPDirectoryProvider - getUser() failed [eid:
> 5804290142082]
> LDAPException: Client request timed out (85) LDAP Timeout
> at com.novell.ldap.Message$Timeout.run(Unknown Source)
> 2013-11-01 09:45:29,600 ERROR http-bio-8080-exec-284
> edu.amc.sakai.user.JLDAPDirectoryProvider - getUser() failed [eid:
> 5911295089083]
> LDAPException: Client request timed out (85) LDAP Timeout
> at com.novell.ldap.Message$Timeout.run(Unknown Source)
> 2013-11-01 09:45:34,406 WARN http-bio-8080-exec-302
> org.sakaiproject.portal.util.ErrorReporter - Bug Report bug-id:
> 4b8c41bb-ec01-41ed-8186-02448c497c6c user: null usage-session: null time:
> 01-Nov-2013 09:45:29 user comment: null stack trace
> org.sakaiproject.portal.api.PortalHandlerException:
> java.lang.RuntimeException: authenticateUser(): LDAPException during
> authentication attempt [userLogin = support][result code = LDAP
> Timeout][error message = null]
> at
> org.sakaiproject.portal.charon.handlers.XLoginHandler.doPost(XLoginHandler.java:50)
> caused by: java.lang.RuntimeException: authenticateUser(): LDAPException
> during authentication attempt [userLogin = support][result code = LDAP
> Timeout][error message = null]
> at
> org.sakaiproject.user.impl.BaseUserDirectoryService.getProviderAuthenticatedUser(BaseUserDirectoryService.java:1597)
> caused by: LDAPException: Client request timed out (85) LDAP Timeout
> at com.novell.ldap.Message$Timeout.run(null:-1)
>
> Tool Placement:
> No Placement
>
> Request:
> : AuthType:null
> : CharEncoding:UTF-8
> : ContentLength:33
> : ContentType:application/x-www-form-urlencoded
> : ContextPath:/portal
> : LocalAddress:192.168.20.5
> : LocalName:opportunity-10.da.org.za
> : LocalPort:8080
> : Method:POST
> : PathInfo:/xlogin
> : Protocol:HTTP/1.1
> : QueryString:null
> : RemoteAddress:46.4.206.124
> : RemoteHost:46.4.206.124
> : RemotePort:56684
> : Requested URL:http://umthombo.da.org.za/portal/xlogin
> : Scheme:http
> : ServerName:umthombo.da.org.za
> : Headers:
> : Header:host:umthombo.da.org.za
> : Header:content-length:33
> :
> Header:accept:text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,*/*;q=0.8
> : Header:origin:http://umthombo.da.org.za
> : Header:user-agent:Mozilla/5.0 (Windows NT 6.2; WOW64)
> AppleWebKit/537.36 (KHTML, like Gecko) Chrome/30.0.1599.101 Safari/537.36
> : Header:content-type:application/x-www-form-urlencoded
> : Header:referer:http://umthombo.da.org.za/portal
> : Header:accept-encoding:gzip,deflate,sdch
> : Header:accept-language:en-US,en;q=0.8
> : Header:cookie:---censored---
> : Header:via:1.1 proxy.iuncapped.co.za (squid/3.1.10)
> : Header:x-forwarded-for:192.168.35.1
> : Header:x-forwarded-for:41.216.135.90
> : Header:cache-control:max-age=259200
> : Header:connection:keep-alive
> : Parameters:
> : Parameter:eid:0:----censored----
> : Parameter:pw:0:----censored----
> : Parameter:submit:0:Login
> : Attributes:
> :
> Attribute:sakai.character.encoding.done:sakai.character.encoding.done
> : Attribute:sakai.filtered:sakai.filtered
> :
> Attribute:sakai.session:org.sakaiproject.tool.impl.MySession at 20cce106
> Session:
> : Created:1383290355854
> : LastAccess:1383291929601
> : MaxInactive:3600
> : Attributes:
> : Attribute:portalskin:neoskin
>
> 2013-11-01 09:45:34,406 WARN http-bio-8080-exec-302
> org.sakaiproject.email.impl.BasicEmailService - send:
> javax.mail.internet.AddressException: Illegal address in string ``''
> Exception in thread "Thread-12" java.lang.OutOfMemoryError: Java heap space
>
>
> -- Warwick Bruce Chapman | +27 83 7797 094 | http://warwickchapman.com
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://collab.sakaiproject.org/pipermail/sakai-dev/attachments/20131101/d7e7a2cd/attachment.html
More information about the sakai-dev
mailing list