[Building Sakai] Application Servers Crashing
David Wafula Wanyonyi
DavidWafula.Wanyonyi at wits.ac.za
Fri Nov 1 02:22:19 PDT 2013
We use Solaris Zones.
Thanks
________________________________
From: Warwick Chapman [warwickchapman at gmail.com]
Sent: Friday, November 01, 2013 11:04 AM
To: David Wafula Wanyonyi
Cc: sakai-dev
Subject: Re: [Building Sakai] Application Servers Crashing
Ps. Are you using OpenVZ as well?
-- Warwick Bruce Chapman | +27 83 7797 094 | http://warwickchapman.com
On Fri, Nov 1, 2013 at 10:52 AM, David Wafula Wanyonyi <DavidWafula.Wanyonyi at wits.ac.za<mailto:DavidWafula.Wanyonyi at wits.ac.za>> wrote:
Hi,
We run servers of almost the same specs, we use the following options and they work just fine with way higher number of sessions:
JAVA_OPTS="$JAVA_OPTS -server -d64 -Xmx4g -XX:PermSize=512m -XX:MaxPermSize=1024m -XX:NewSize=512m -XX:MaxNewSize=1024m -Djava.awt.headless=true -Dcom.sun.management.jmxremote -Dsun.lang.ClassLoader.allowArraySyntax=true -Djava.util.Arrays.useLegacyMergeSort=true"
You might want up your PermSize.
Regards,
david.
________________________________
From: Warwick Chapman [warwickchapman at gmail.com<mailto:warwickchapman at gmail.com>]
Sent: Friday, November 01, 2013 10:27 AM
To: sakai-dev
Subject: Re: [Building Sakai] Application Servers Crashing
The three are running at the moment, with 63 sessions shared between them:
root at opportunity:~# uname -a
Linux opportunity 2.6.32-19-pve #1 SMP Wed May 15 07:32:52 CEST 2013 x86_64 x86_64 x86_64 GNU/Linux
root at opportunity:~# free -m
total used free shared buffers cached
Mem: 4096 3679 416 0 0 1968
-/+ buffers/cache: 1711 2384
Swap: 4096 6 4089
root at opportunity:~# w
10:23:19 up 1 day, 14:23, 1 user, load average: 0,16, 0,08, 0,04
---
root at opportunity-01:~# uname -a
Linux opportunity-01 2.6.32-19-pve #1 SMP Wed May 15 07:32:52 CEST 2013 x86_64 x86_64 x86_64 GNU/Linux
root at opportunity-01:~# free -m
total used free shared buffers cached
Mem: 4096 2662 1433 0 0 954
-/+ buffers/cache: 1708 2387
Swap: 4096 3 4092
root at opportunity-01:~# w
10:23:54 up 22:56, 1 user, load average: 0,03, 0,05, 0,00
---
root at opportunity-10:~# uname -a
Linux opportunity-10 2.6.32-20-pve #1 SMP Wed May 15 08:23:27 CEST 2013 x86_64 x86_64 x86_64 GNU/Linux
root at opportunity-10:~# free -m
total used free shared buffers cached
Mem: 4096 1778 2317 0 0 146
-/+ buffers/cache: 1632 2463
Swap: 4096 0 4096
root at opportunity-10:~# w
10:24:40 up 10:40, 2 users, load average: 0,00, 0,02, 0,13
-- Warwick Bruce Chapman | +27 83 7797 094<tel:%2B27%2083%207797%20094> | http://warwickchapman.com
On Fri, Nov 1, 2013 at 10:04 AM, Warwick Chapman <warwickchapman at gmail.com<mailto:warwickchapman at gmail.com>> wrote:
Hi All
I'm going mad here. Surely we can expect a four processor, 4G appserver to handle more than 50 sessions without crashing?
Mine keep crashing and its causing havoc.
As they approach 40 sessions, the system load which is usually below 0.10 start climbing, and then hits 1.00 and eventually I get the "java.lang.OutOfMemoryError: Java heap space" error.
Details
1. Ubuntu 12.04 OpenVZ on Proxmox with 4G RAM and 4 processors allocated.
2. Oracle Java 6:
root at opportunity-10:~# java -version
java version "1.6.0_45"
Java(TM) SE Runtime Environment (build 1.6.0_45-b06)
Java HotSpot(TM) 64-Bit Server VM (build 20.45-b01, mixed mode)
3. Environment variables:
root at opportunity-10:~# set | grep JAVA
JAVA_HOME=/usr/lib/jvm/java-6-oracle
JAVA_OPTS='-server -Xms512m -Xmx1024m -XX:PermSize=128m -XX:MaxPermSize=512m -XX:NewSize=192m -XX:MaxNewSize=384m -Djava.awt.headless=true -Dhttp.agent=Sakai -Dorg.apache.jasper.compiler.Parser.STRICT_QUOTE_ESCAPING=false -Dsun.lang.ClassLoader.allowArraySyntax=true -Duser.language=en -Duser.region=ZA -Dfile.encoding=UTF-8'
4. Sakai 2.9.3 built independently on each app server (also tried building on one and deploying to others). Custom providers, SMS and only patch is for UCT SMS.
5. Process:
31032 pts/2 Sl 1:44 /usr/lib/jvm/java-6-oracle/bin/java -Djava.util.logging.config.file=/opt/tomcat/conf/logging.properties -Djava.util.logging.manager=org.apache.juli.ClassLoaderLogManager -server -Xms512m -Xmx1024m -XX:PermSize=128m -XX:MaxPermSize=512m -XX:NewSize=192m -XX:MaxNewSize=384m -Djava.awt.headless=true -Dhttp.agent=Sakai -Dorg.apache.jasper.compiler.Parser.STRICT_QUOTE_ESCAPING=false -Dsun.lang.ClassLoader.allowArraySyntax=true -Duser.language=en -Duser.region=ZA -Dfile.encoding=UTF-8 -Djava.endorsed.dirs=/opt/tomcat/endorsed -classpath /opt/tomcat/bin/bootstrap.jar:/opt/tomcat/bin/tomcat-juli.jar -Dcatalina.base=/opt/tomcat -Dcatalina.home=/opt/tomcat -Djava.io.tmpdir=/opt/tomcat/temp org.apache.catalina.startup.Bootstrap start
6. Database server and LDAP server are quiet and not battling with load at all.
7. The underlying host machines are quiet and load is not affecting them at all and memory usage is safely below available.
See catalina for the most recent crash:
2013-11-01 09:37:18,497 WARN http-bio-8080-exec-281 org.sakaiproject.portal.charon.site.DefaultSiteViewImpl - Failed to set canAddSite for current user. Defaulting to false ...
Exception in thread "PageSavingThread-sakai.profile2" java.lang.OutOfMemoryError: Java heap space
2013-11-01 09:41:04,553 ERROR http-bio-8080-exec-256 org.hibernate.transaction.JDBCTransaction - JDBC rollback failed
com.mysql.jdbc.exceptions.jdbc4.MySQLNonTransientConnectionException: No operations allowed after connection closed.
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:39)
at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:27)
at java.lang.reflect.Constructor.newInstance(Constructor.java:513)
at com.mysql.jdbc.Util.handleNewInstance(Util.java:411)
at com.mysql.jdbc.Util.getInstance(Util.java:386)
at com.mysql.jdbc.SQLError.createSQLException(SQLError.java:1015)
at com.mysql.jdbc.SQLError.createSQLException(SQLError.java:989)
at com.mysql.jdbc.SQLError.createSQLException(SQLError.java:975)
at com.mysql.jdbc.SQLError.createSQLException(SQLError.java:920)
at com.mysql.jdbc.ConnectionImpl.throwConnectionClosedException(ConnectionImpl.java:1304)
at com.mysql.jdbc.ConnectionImpl.checkClosed(ConnectionImpl.java:1296)
at com.mysql.jdbc.ConnectionImpl.rollback(ConnectionImpl.java:5028)
at org.apache.commons.dbcp.DelegatingConnection.rollback(DelegatingConnection.java:328)
at org.apache.commons.dbcp.PoolingDataSource$PoolGuardConnectionWrapper.rollback(PoolingDataSource.java:312)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at org.springframework.jdbc.datasource.LazyConnectionDataSourceProxy$LazyConnectionInvocationHandler.invoke(LazyConnectionDataSourceProxy.java:364)
at com.sun.proxy.$Proxy5.rollback(Unknown Source)
at org.hibernate.transaction.JDBCTransaction.rollbackAndResetAutoCommit(JDBCTransaction.java:183)
at org.hibernate.transaction.JDBCTransaction.rollback(JDBCTransaction.java:162)
at org.springframework.orm.hibernate3.HibernateTransactionManager.doRollback(HibernateTransactionManager.java:674)
at org.springframework.transaction.support.AbstractPlatformTransactionManager.processRollback(AbstractPlatformTransactionManager.java:823)
at org.springframework.transaction.support.AbstractPlatformTransactionManager.rollback(AbstractPlatformTransactionManager.java:800)
at org.springframework.transaction.interceptor.TransactionAspectSupport.completeTransactionAfterThrowing(TransactionAspectSupport.java:339)
at org.springframework.transaction.interceptor.TransactionInterceptor.invoke(TransactionInterceptor.java:110)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:171)
at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:204)
at com.sun.proxy.$Proxy42.getPublishedAssessment(Unknown Source)
at org.sakaiproject.tool.assessment.services.assessment.PublishedAssessmentService.getPublishedAssessment(PublishedAssessmentService.java:189)
at org.sakaiproject.tool.assessment.ui.listener.delivery.BeginDeliveryActionListener.lookupPublishedAssessment(BeginDeliveryActionListener.java:161)
at org.sakaiproject.tool.assessment.ui.listener.delivery.BeginDeliveryActionListener.getPublishedAssessmentBasedOnAction(BeginDeliveryActionListener.java:415)
at org.sakaiproject.tool.assessment.ui.listener.delivery.BeginDeliveryActionListener.processAction(BeginDeliveryActionListener.java:109)
at javax.faces.event.ActionEvent.processListener(ActionEvent.java:77)
at javax.faces.component.UIComponentBase.broadcast(UIComponentBase.java:669)
at javax.faces.component.UICommand.broadcast(UICommand.java:317)
at javax.faces.component.UIData.broadcast(UIData.java:677)
at javax.faces.component.UIViewRoot.broadcastEvents(UIViewRoot.java:287)
at javax.faces.component.UIViewRoot.processApplication(UIViewRoot.java:401)
at com.sun.faces.lifecycle.InvokeApplicationPhase.execute(InvokeApplicationPhase.java:95)
at com.sun.faces.lifecycle.LifecycleImpl.phase(LifecycleImpl.java:245)
at com.sun.faces.lifecycle.LifecycleImpl.execute(LifecycleImpl.java:110)
at javax.faces.webapp.FacesServlet.service(FacesServlet.java:213)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:305)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:210)
at org.sakaiproject.util.RequestFilter.doFilter(RequestFilter.java:634)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:243)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:210)
at org.apache.catalina.core.ApplicationDispatcher.invoke(ApplicationDispatcher.java:749)
at org.apache.catalina.core.ApplicationDispatcher.processRequest(ApplicationDispatcher.java:487)
at org.apache.catalina.core.ApplicationDispatcher.doForward(ApplicationDispatcher.java:412)
at org.apache.catalina.core.ApplicationDispatcher.forward(ApplicationDispatcher.java:339)
at org.sakaiproject.jsf.util.SamigoJsfTool.dispatch(SamigoJsfTool.java:301)
at org.sakaiproject.jsf.util.JsfTool.doPost(JsfTool.java:256)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:647)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:728)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:305)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:210)
at org.apache.catalina.core.ApplicationDispatcher.invoke(ApplicationDispatcher.java:749)
at org.apache.catalina.core.ApplicationDispatcher.processRequest(ApplicationDispatcher.java:487)
at org.apache.catalina.core.ApplicationDispatcher.doForward(ApplicationDispatcher.java:379)
at org.apache.catalina.core.ApplicationDispatcher.forward(ApplicationDispatcher.java:339)
at org.sakaiproject.tool.impl.ActiveToolComponent$MyActiveTool.forward(ActiveToolComponent.java:511)
at org.sakaiproject.portal.charon.SkinnableCharonPortal.forwardTool(SkinnableCharonPortal.java:1483)
at org.sakaiproject.portal.charon.handlers.ToolHandler.doTool(ToolHandler.java:213)
at org.sakaiproject.portal.charon.handlers.ToolHandler.doGet(ToolHandler.java:96)
at org.sakaiproject.portal.charon.handlers.ToolHandler.doPost(ToolHandler.java:73)
at org.sakaiproject.portal.charon.SkinnableCharonPortal.doPost(SkinnableCharonPortal.java:1273)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:647)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:728)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:305)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:210)
at org.sakaiproject.util.RequestFilter.doFilter(RequestFilter.java:695)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:243)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:210)
at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:222)
at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:123)
at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:502)
at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:171)
at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:100)
at org.apache.catalina.valves.AccessLogValve.invoke(AccessLogValve.java:953)
at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:118)
at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:408)
at org.apache.coyote.http11.AbstractHttp11Processor.process(AbstractHttp11Processor.java:1041)
at org.apache.coyote.AbstractProtocol$AbstractConnectionHandler.process(AbstractProtocol.java:603)
at org.apache.tomcat.util.net.JIoEndpoint$SocketProcessor.run(JIoEndpoint.java:310)
at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:895)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:918)
at java.lang.Thread.run(Thread.java:662)
Caused by: java.lang.OutOfMemoryError: Java heap space
2013-11-01 09:41:14,635 WARN SakaiClusterService.Maintenance org.sakaiproject.db.impl.BasicSqlService - Sql.dbRead: sql: select SERVER_ID from SAKAI_CLUSTER where SERVER_ID = ? opportunity-10-1383259498265
com.mysql.jdbc.exceptions.jdbc4.CommunicationsException: Communications link failure
The last packet successfully received from the server was 6,629 milliseconds ago. The last packet sent successfully to the server was 6,629 milliseconds ago.
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:39)
at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:27)
at java.lang.reflect.Constructor.newInstance(Constructor.java:513)
at com.mysql.jdbc.Util.handleNewInstance(Util.java:411)
at com.mysql.jdbc.SQLError.createCommunicationsException(SQLError.java:1121)
at com.mysql.jdbc.MysqlIO.reuseAndReadPacket(MysqlIO.java:3673)
at com.mysql.jdbc.MysqlIO.reuseAndReadPacket(MysqlIO.java:3562)
at com.mysql.jdbc.MysqlIO.checkErrorPacket(MysqlIO.java:4113)
at com.mysql.jdbc.MysqlIO.checkErrorPacket(MysqlIO.java:927)
at com.mysql.jdbc.MysqlIO.secureAuth411(MysqlIO.java:4689)
at com.mysql.jdbc.MysqlIO.doHandshake(MysqlIO.java:1304)
at com.mysql.jdbc.ConnectionImpl.coreConnect(ConnectionImpl.java:2486)
at com.mysql.jdbc.ConnectionImpl.connectOneTryOnly(ConnectionImpl.java:2519)
at com.mysql.jdbc.ConnectionImpl.createNewIO(ConnectionImpl.java:2304)
at com.mysql.jdbc.ConnectionImpl.<init>(ConnectionImpl.java:834)
at com.mysql.jdbc.JDBC4Connection.<init>(JDBC4Connection.java:47)
at sun.reflect.GeneratedConstructorAccessor680.newInstance(Unknown Source)
at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:27)
at java.lang.reflect.Constructor.newInstance(Constructor.java:513)
at com.mysql.jdbc.Util.handleNewInstance(Util.java:411)
at com.mysql.jdbc.ConnectionImpl.getInstance(ConnectionImpl.java:416)
at com.mysql.jdbc.NonRegisteringDriver.connect(NonRegisteringDriver.java:346)
at org.apache.commons.dbcp.DriverConnectionFactory.createConnection(DriverConnectionFactory.java:38)
at org.apache.commons.dbcp.PoolableConnectionFactory.makeObject(PoolableConnectionFactory.java:294)
at org.apache.commons.pool.impl.GenericObjectPool.borrowObject(GenericObjectPool.java:1181)
at org.apache.commons.dbcp.PoolingDataSource.getConnection(PoolingDataSource.java:96)
at org.apache.commons.dbcp.BasicDataSource.getConnection(BasicDataSource.java:880)
at org.sakaiproject.db.impl.BasicSqlService.borrowConnection(BasicSqlService.java:260)
at org.sakaiproject.db.impl.BasicSqlService.dbRead(BasicSqlService.java:540)
at org.sakaiproject.db.impl.BasicSqlService.dbRead(BasicSqlService.java:471)
at org.sakaiproject.cluster.impl.SakaiClusterService$Maintenance.run(SakaiClusterService.java:377)
at java.lang.Thread.run(Thread.java:662)
Caused by: java.io.EOFException: Can not read response from server. Expected to read 4 bytes, read 0 bytes before connection was unexpectedly lost.
at com.mysql.jdbc.MysqlIO.readFully(MysqlIO.java:3119)
at com.mysql.jdbc.MysqlIO.reuseAndReadPacket(MysqlIO.java:3573)
... 26 more
2013-11-01 09:41:27,726 WARN SakaiClusterService.Maintenance org.sakaiproject.cluster.impl.SakaiClusterService - run(): server has been closed in cluster table, reopened: opportunity-10-1383259498265
2013-11-01 09:41:24,623 WARN http-bio-8080-exec-291 org.sakaiproject.portal.util.ErrorReporter - Bug Report bug-id: 1c74337e-3721-4fb6-812d-85f665ff1da1 user: 334d3096-e735-4597-9826-a1f6fcb44d1a usage-session: 9dbb01fb-9a47-4435-b019-3c626f9ec35c time: 01-Nov-2013 09:39:42 user comment: null stack trace
java.lang.OutOfMemoryError: Java heap space
at java.util.Hashtable.<init>(Hashtable.java:162)
at java.util.Hashtable.<init>(Hashtable.java:183)
at org.sakaiproject.util.BaseResourceProperties.<init>(BaseResourceProperties.java:82)
at org.sakaiproject.util.BaseResourcePropertiesEdit.<init>(BaseResourcePropertiesEdit.java:39)
at org.sakaiproject.site.impl.BaseSite.set(BaseSite.java:575)
at org.sakaiproject.site.impl.BaseSite.<init>(BaseSite.java:234)
at org.sakaiproject.site.impl.BaseSiteService.getCachedSite(BaseSiteService.java:599)
at org.sakaiproject.site.impl.BaseSiteService.getDefinedSite(BaseSiteService.java:620)
at org.sakaiproject.site.impl.BaseSiteService.getSite(BaseSiteService.java:709)
at org.sakaiproject.site.impl.BaseSiteService.getSiteVisit(BaseSiteService.java:787)
at org.sakaiproject.site.cover.SiteService.getSiteVisit(SiteService.java:140)
at org.sakaiproject.portal.charon.handlers.ToolHandler.doTool(ToolHandler.java:167)
at org.sakaiproject.portal.charon.handlers.ToolHandler.doGet(ToolHandler.java:96)
at org.sakaiproject.portal.charon.SkinnableCharonPortal.doGet(SkinnableCharonPortal.java:894)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:621)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:728)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:305)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:210)
at org.sakaiproject.util.RequestFilter.doFilter(RequestFilter.java:695)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:243)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:210)
at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:222)
at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:123)
at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:502)
at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:171)
at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:100)
at org.apache.catalina.valves.AccessLogValve.invoke(AccessLogValve.java:953)
at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:118)
at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:408)
at org.apache.coyote.http11.AbstractHttp11Processor.process(AbstractHttp11Processor.java:1041)
at org.apache.coyote.AbstractProtocol$AbstractConnectionHandler.process(AbstractProtocol.java:603)
at org.apache.tomcat.util.net.JIoEndpoint$SocketProcessor.run(JIoEndpoint.java:312)
Tool Placement:
No Placement
Request:
: AuthType:null
: CharEncoding:UTF-8
: ContentLength:-1
: ContentType:null
: ContextPath:/portal
: LocalAddress:192.168.20.5
: LocalName:opportunity-10.da.org.za<http://opportunity-10.da.org.za>
: LocalPort:8080
: Method:GET
: PathInfo:/tool/00c1784d-3cb9-4b9f-ada5-4200b82376db/synMainLite
: Protocol:HTTP/1.1
: QueryString:time=1
: RemoteAddress:46.4.206.124
: RemoteHost:46.4.206.124
: RemotePort:56498
: Requested URL:http://umthombo.da.org.za/portal/tool/00c1784d-3cb9-4b9f-ada5-4200b82376db/synMainLite
: Scheme:http
: ServerName:umthombo.da.org.za<http://umthombo.da.org.za>
: Headers:
: Header:accept:text/html, application/xhtml+xml, */*
: Header:referer:http://umthombo.da.org.za/portal/tool/00c1784d-3cb9-4b9f-ada5-4200b82376db/wait
: Header:accept-language:en-ZA
: Header:user-agent:Mozilla/5.0 (compatible; MSIE 10.0; Windows NT 6.1; WOW64; Trident/6.0)
: Header:accept-encoding:gzip, deflate
: Header:dnt:1
: Header:cookie:---censored---
: Header:host:umthombo.da.org.za<http://umthombo.da.org.za>
: Header:cache-control:max-age=43200
: Header:connection:keep-alive
: Header:x-forwarded-for:41.13.48.159
: Parameters:
: Parameter:time:0:1
: Attributes:
: Attribute:sakai.character.encoding.done:sakai.character.encoding.done
: Attribute:sakai.filtered:sakai.filtered
: Attribute:sakai.session:org.sakaiproject.tool.impl.MySession at a064900
Session:
: Created:1383291288806
: LastAccess:1383291671451
: MaxInactive:3600
: Attributes:
: Attribute:portalskin:neoskin
2013-11-01 09:41:35,971 WARN http-bio-8080-exec-256 org.hibernate.util.JDBCExceptionReporter - SQL Error: 0, SQLState: null
2013-11-01 09:41:35,971 ERROR http-bio-8080-exec-256 org.hibernate.util.JDBCExceptionReporter - Already closed.
2013-11-01 09:42:09,783 WARN http-bio-8080-exec-256 org.hibernate.util.JDBCExceptionReporter - SQL Error: 0, SQLState: null
2013-11-01 09:42:09,783 ERROR http-bio-8080-exec-256 org.hibernate.util.JDBCExceptionReporter - Already closed.
2013-11-01 09:45:04,447 ERROR http-bio-8080-exec-256 org.springframework.transaction.interceptor.TransactionInterceptor - Application exception overridden by rollback exception
java.lang.OutOfMemoryError: Java heap space
2013-11-01 09:45:04,447 ERROR http-bio-8080-exec-256 org.sakaiproject.tool.assessment.services.assessment.PublishedAssessmentService - org.springframework.transaction.TransactionSystemException: Could not roll back Hibernate transaction; nested exception is org.hibernate.TransactionException: JDBC rollback failed
2013-11-01 09:45:04,447 ERROR http-bio-8080-exec-256 com.sun.faces.lifecycle.InvokeApplicationPhase - org.springframework.transaction.TransactionSystemException: Could not roll back Hibernate transaction; nested exception is org.hibernate.TransactionException: JDBC rollback failed
java.lang.RuntimeException: org.springframework.transaction.TransactionSystemException: Could not roll back Hibernate transaction; nested exception is org.hibernate.TransactionException: JDBC rollback failed
at org.sakaiproject.tool.assessment.services.assessment.PublishedAssessmentService.getPublishedAssessment(PublishedAssessmentService.java:195)
at org.sakaiproject.tool.assessment.ui.listener.delivery.BeginDeliveryActionListener.lookupPublishedAssessment(BeginDeliveryActionListener.java:161)
at org.sakaiproject.tool.assessment.ui.listener.delivery.BeginDeliveryActionListener.getPublishedAssessmentBasedOnAction(BeginDeliveryActionListener.java:415)
at org.sakaiproject.tool.assessment.ui.listener.delivery.BeginDeliveryActionListener.processAction(BeginDeliveryActionListener.java:109)
at javax.faces.event.ActionEvent.processListener(ActionEvent.java:77)
at javax.faces.component.UIComponentBase.broadcast(UIComponentBase.java:669)
at javax.faces.component.UICommand.broadcast(UICommand.java:317)
at javax.faces.component.UIData.broadcast(UIData.java:677)
at javax.faces.component.UIViewRoot.broadcastEvents(UIViewRoot.java:287)
at javax.faces.component.UIViewRoot.processApplication(UIViewRoot.java:401)
at com.sun.faces.lifecycle.InvokeApplicationPhase.execute(InvokeApplicationPhase.java:95)
at com.sun.faces.lifecycle.LifecycleImpl.phase(LifecycleImpl.java:245)
at com.sun.faces.lifecycle.LifecycleImpl.execute(LifecycleImpl.java:110)
at javax.faces.webapp.FacesServlet.service(FacesServlet.java:213)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:305)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:210)
at org.sakaiproject.util.RequestFilter.doFilter(RequestFilter.java:634)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:243)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:210)
at org.apache.catalina.core.ApplicationDispatcher.invoke(ApplicationDispatcher.java:749)
at org.apache.catalina.core.ApplicationDispatcher.processRequest(ApplicationDispatcher.java:487)
at org.apache.catalina.core.ApplicationDispatcher.doForward(ApplicationDispatcher.java:412)
at org.apache.catalina.core.ApplicationDispatcher.forward(ApplicationDispatcher.java:339)
at org.sakaiproject.jsf.util.SamigoJsfTool.dispatch(SamigoJsfTool.java:301)
at org.sakaiproject.jsf.util.JsfTool.doPost(JsfTool.java:256)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:647)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:728)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:305)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:210)
at org.apache.catalina.core.ApplicationDispatcher.invoke(ApplicationDispatcher.java:749)
at org.apache.catalina.core.ApplicationDispatcher.processRequest(ApplicationDispatcher.java:487)
at org.apache.catalina.core.ApplicationDispatcher.doForward(ApplicationDispatcher.java:379)
at org.apache.catalina.core.ApplicationDispatcher.forward(ApplicationDispatcher.java:339)
at org.sakaiproject.tool.impl.ActiveToolComponent$MyActiveTool.forward(ActiveToolComponent.java:511)
at org.sakaiproject.portal.charon.SkinnableCharonPortal.forwardTool(SkinnableCharonPortal.java:1483)
at org.sakaiproject.portal.charon.handlers.ToolHandler.doTool(ToolHandler.java:213)
at org.sakaiproject.portal.charon.handlers.ToolHandler.doGet(ToolHandler.java:96)
at org.sakaiproject.portal.charon.handlers.ToolHandler.doPost(ToolHandler.java:73)
at org.sakaiproject.portal.charon.SkinnableCharonPortal.doPost(SkinnableCharonPortal.java:1273)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:647)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:728)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:305)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:210)
at org.sakaiproject.util.RequestFilter.doFilter(RequestFilter.java:695)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:243)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:210)
at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:222)
at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:123)
at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:502)
at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:171)
at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:100)
at org.apache.catalina.valves.AccessLogValve.invoke(AccessLogValve.java:953)
at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:118)
at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:408)
at org.apache.coyote.http11.AbstractHttp11Processor.process(AbstractHttp11Processor.java:1041)
at org.apache.coyote.AbstractProtocol$AbstractConnectionHandler.process(AbstractProtocol.java:603)
at org.apache.tomcat.util.net.JIoEndpoint$SocketProcessor.run(JIoEndpoint.java:310)
at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:895)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:918)
at java.lang.Thread.run(Thread.java:662)
Caused by: org.springframework.transaction.TransactionSystemException: Could not roll back Hibernate transaction; nested exception is org.hibernate.TransactionException: JDBC rollback failed
at org.springframework.orm.hibernate3.HibernateTransactionManager.doRollback(HibernateTransactionManager.java:677)
at org.springframework.transaction.support.AbstractPlatformTransactionManager.processRollback(AbstractPlatformTransactionManager.java:823)
at org.springframework.transaction.support.AbstractPlatformTransactionManager.rollback(AbstractPlatformTransactionManager.java:800)
at org.springframework.transaction.interceptor.TransactionAspectSupport.completeTransactionAfterThrowing(TransactionAspectSupport.java:339)
at org.springframework.transaction.interceptor.TransactionInterceptor.invoke(TransactionInterceptor.java:110)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:171)
at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:204)
at com.sun.proxy.$Proxy42.getPublishedAssessment(Unknown Source)
at org.sakaiproject.tool.assessment.services.assessment.PublishedAssessmentService.getPublishedAssessment(PublishedAssessmentService.java:189)
... 59 more
Caused by: org.hibernate.TransactionException: JDBC rollback failed
at org.hibernate.transaction.JDBCTransaction.rollback(JDBCTransaction.java:170)
at org.springframework.orm.hibernate3.HibernateTransactionManager.doRollback(HibernateTransactionManager.java:674)
... 67 more
Caused by: com.mysql.jdbc.exceptions.jdbc4.MySQLNonTransientConnectionException: No operations allowed after connection closed.
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:39)
at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:27)
at java.lang.reflect.Constructor.newInstance(Constructor.java:513)
at com.mysql.jdbc.Util.handleNewInstance(Util.java:411)
at com.mysql.jdbc.Util.getInstance(Util.java:386)
at com.mysql.jdbc.SQLError.createSQLException(SQLError.java:1015)
at com.mysql.jdbc.SQLError.createSQLException(SQLError.java:989)
at com.mysql.jdbc.SQLError.createSQLException(SQLError.java:975)
at com.mysql.jdbc.SQLError.createSQLException(SQLError.java:920)
at com.mysql.jdbc.ConnectionImpl.throwConnectionClosedException(ConnectionImpl.java:1304)
at com.mysql.jdbc.ConnectionImpl.checkClosed(ConnectionImpl.java:1296)
at com.mysql.jdbc.ConnectionImpl.rollback(ConnectionImpl.java:5028)
at org.apache.commons.dbcp.DelegatingConnection.rollback(DelegatingConnection.java:328)
at org.apache.commons.dbcp.PoolingDataSource$PoolGuardConnectionWrapper.rollback(PoolingDataSource.java:312)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at org.springframework.jdbc.datasource.LazyConnectionDataSourceProxy$LazyConnectionInvocationHandler.invoke(LazyConnectionDataSourceProxy.java:364)
at com.sun.proxy.$Proxy5.rollback(Unknown Source)
at org.hibernate.transaction.JDBCTransaction.rollbackAndResetAutoCommit(JDBCTransaction.java:183)
at org.hibernate.transaction.JDBCTransaction.rollback(JDBCTransaction.java:162)
... 68 more
Caused by: java.lang.OutOfMemoryError: Java heap space
2013-11-01 09:45:29,599 ERROR http-bio-8080-exec-295 edu.amc.sakai.user.JLDAPDirectoryProvider - getUser() failed [eid: 7306145161083]
LDAPException: Client request timed out (85) LDAP Timeout
at com.novell.ldap.Message$Timeout.run(Unknown Source)
2013-11-01 09:45:29,600 ERROR http-bio-8080-exec-289 edu.amc.sakai.user.JLDAPDirectoryProvider - getUser() failed [eid: 5804290142082]
LDAPException: Client request timed out (85) LDAP Timeout
at com.novell.ldap.Message$Timeout.run(Unknown Source)
2013-11-01 09:45:29,600 ERROR http-bio-8080-exec-284 edu.amc.sakai.user.JLDAPDirectoryProvider - getUser() failed [eid: 5911295089083]
LDAPException: Client request timed out (85) LDAP Timeout
at com.novell.ldap.Message$Timeout.run(Unknown Source)
2013-11-01 09:45:34,406 WARN http-bio-8080-exec-302 org.sakaiproject.portal.util.ErrorReporter - Bug Report bug-id: 4b8c41bb-ec01-41ed-8186-02448c497c6c user: null usage-session: null time: 01-Nov-2013 09:45:29 user comment: null stack trace
org.sakaiproject.portal.api.PortalHandlerException: java.lang.RuntimeException: authenticateUser(): LDAPException during authentication attempt [userLogin = support][result code = LDAP Timeout][error message = null]
at org.sakaiproject.portal.charon.handlers.XLoginHandler.doPost(XLoginHandler.java:50)
caused by: java.lang.RuntimeException: authenticateUser(): LDAPException during authentication attempt [userLogin = support][result code = LDAP Timeout][error message = null]
at org.sakaiproject.user.impl.BaseUserDirectoryService.getProviderAuthenticatedUser(BaseUserDirectoryService.java:1597)
caused by: LDAPException: Client request timed out (85) LDAP Timeout
at com.novell.ldap.Message$Timeout.run(null:-1)
Tool Placement:
No Placement
Request:
: AuthType:null
: CharEncoding:UTF-8
: ContentLength:33
: ContentType:application/x-www-form-urlencoded
: ContextPath:/portal
: LocalAddress:192.168.20.5
: LocalName:opportunity-10.da.org.za<http://opportunity-10.da.org.za>
: LocalPort:8080
: Method:POST
: PathInfo:/xlogin
: Protocol:HTTP/1.1
: QueryString:null
: RemoteAddress:46.4.206.124
: RemoteHost:46.4.206.124
: RemotePort:56684
: Requested URL:http://umthombo.da.org.za/portal/xlogin
: Scheme:http
: ServerName:umthombo.da.org.za<http://umthombo.da.org.za>
: Headers:
: Header:host:umthombo.da.org.za<http://umthombo.da.org.za>
: Header:content-length:33
: Header:accept:text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,*/*;q=0.8
: Header:origin:http://umthombo.da.org.za
: Header:user-agent:Mozilla/5.0 (Windows NT 6.2; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/30.0.1599.101 Safari/537.36
: Header:content-type:application/x-www-form-urlencoded
: Header:referer:http://umthombo.da.org.za/portal
: Header:accept-encoding:gzip,deflate,sdch
: Header:accept-language:en-US,en;q=0.8
: Header:cookie:---censored---
: Header:via:1.1 proxy.iuncapped.co.za<http://proxy.iuncapped.co.za> (squid/3.1.10)
: Header:x-forwarded-for:192.168.35.1
: Header:x-forwarded-for:41.216.135.90
: Header:cache-control:max-age=259200
: Header:connection:keep-alive
: Parameters:
: Parameter:eid:0:----censored----
: Parameter:pw:0:----censored----
: Parameter:submit:0:Login
: Attributes:
: Attribute:sakai.character.encoding.done:sakai.character.encoding.done
: Attribute:sakai.filtered:sakai.filtered
: Attribute:sakai.session:org.sakaiproject.tool.impl.MySession at 20cce106
Session:
: Created:1383290355854
: LastAccess:1383291929601
: MaxInactive:3600
: Attributes:
: Attribute:portalskin:neoskin
2013-11-01 09:45:34,406 WARN http-bio-8080-exec-302 org.sakaiproject.email.impl.BasicEmailService - send: javax.mail.internet.AddressException: Illegal address in string ``''
Exception in thread "Thread-12" java.lang.OutOfMemoryError: Java heap space
-- Warwick Bruce Chapman | +27 83 7797 094<tel:%2B27%2083%207797%20094> | http://warwickchapman.com
This communication is intended
for the addressee only. It is confidential. If you have received this communication in error, please notify us immediately and destroy the original message. You may not copy or disseminate this communication without the permission of the University. Only authorised
signatories are competent to enter into agreements on behalf of the University and recipients are thus advised that the content of this message may not be legally binding on the University and may contain the personal views and opinions of the author, which
are not necessarily the views and opinions of The University of the Witwatersrand, Johannesburg. All agreements between the University and outsiders are subject to South African Law unless the University agrees in writing to the contrary.
<table width="100%" border="0" cellspacing="0" cellpadding="0" style="width:100%;">
<tr>
<td align="left" style="text-align:justify;"><font face="arial,sans-serif" size="1" color="#999999"><span style="font-size:11px;">This communication is intended for the addressee only. It is confidential. If you have received this communication in error, please notify us immediately and destroy the original message. You may not copy or disseminate this communication without the permission of the University. Only authorised signatories are competent to enter into agreements on behalf of the University and recipients are thus advised that the content of this message may not be legally binding on the University and may contain the personal views and opinions of the author, which are not necessarily the views and opinions of The University of the Witwatersrand, Johannesburg. All agreements between the University and outsiders are subject to South African Law unless the University agrees in writing to the contrary. </span></font></td>
</tr>
</table>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://collab.sakaiproject.org/pipermail/sakai-dev/attachments/20131101/3a735624/attachment.html
More information about the sakai-dev
mailing list