[Building Sakai] FW: Session replication fails on ReplicationValve invocation
Gerwood Stewart
gstewar8 at une.edu.au
Wed Jun 17 17:36:06 PDT 2009
This is possibly more of a Tomcat issue but has anyone faced this issue and have they found any work arounds?
Thanks
Gerwood
------ Forwarded Message
From: David Nillesen <dnillese at une.edu.au>
Date: Thu, 18 Jun 2009 10:29:55 +1000
To: Gerwood Stewart <gstewar8 at une.edu.au>
Subject: FW: Session replication fails on ReplicationValve invocation
David Nillesen
UNIX Systems Administrator
University of New England
+61 2 6773 2112
From: David Nillesen
Sent: Thursday, 18 June 2009 10:24 AM
To: 'users at tomcat.apache.org'
Subject: Session replication fails on ReplicationValve invocation
Hi,
I'm currently trying to establish a cluster of tomcat 5.5.27 servers to deploy sakai in, so I'm stuck on version 5.5. It is not an option to upgrade to tomcat 6 which from all accounts has much better session replication.
I have cluster membership up and working via multicast and the two nodes add each other as replication targets. When I try to establish a connection to the server from my web browser to the default port of 8080 I get this neat error in my catalina.out:
ERROR: An exception or error occurred in the container during the request processing (2009-06-17 16:43:49,545 http-172.31.3.37-8080-Processor5_org.apache.catalina.connector.CoyoteAdapter)
java.lang.NullPointerException
at org.apache.catalina.cluster.tcp.ReplicationValve.invoke(ReplicationValve.java:348)
I'll put full errors and configs at the bottom of the email. The errors occurs regardless of whether I access an empty/nonexistent context or the servlet-examples which are currently installed.
The servers are multi homed with 3 interfaces. Eth0 is the primary public interface, eth1 is my iSCSI network and eth2 is the backend cluster network. All neatly VLANd from each other in seperate subnets. Multicast membership works, I can see the packets arriving via tcpdump and the membership propagation is occurring.
I have tried binding the receivers to both public and private interfaces but have had no luck. The receivers are up and appear to be functional. I can telnet to them and when I enter gibberish into the console it tell me I am sending malformed headers.
By using tcpdump and netstat I can see that neither tomcat instance is originating a sender session to connect to the other servers receiver.
I have tried fastsyncqueue and pooled style senders from the documentation. Currently configured for pooled as it looked the simpler config to diagnose. jvmRoute is configured via -D options passed to java as a lot of our configs are done dynamically so as to have a single server.xml across the cluster environments. We have a shared disk backend system to enable this.
Any suggestions for diagnosis, help, fixes or otherwise would be very much appreciated.
Regards,
Dave
Log file and startup:
INFO: The Apache Tomcat Native library which allows optimal performance in production environments was not found on the java.library.path: /apps/java/jdk1.5.0_19/jre/lib/amd64/server:/apps/java/jdk1.5.0_19/jre/lib/amd64:/apps/java/jdk1.5
.0_19/jre/../lib/amd64 (2009-06-17 16:40:36,874 main_org.apache.catalina.core.AprLifecycleListener)
INFO: Initializing Coyote HTTP/1.1 on http-172.31.3.37-8080 (2009-06-17 16:40:37,019 main_org.apache.coyote.http11.Http11BaseProtocol)
INFO: Initialization processed in 1400 ms (2009-06-17 16:40:37,021 main_org.apache.catalina.startup.Catalina)
INFO: Set JAAS app name ENGINE (2009-06-17 16:40:37,114 main_org.apache.catalina.realm.JAASRealm)
INFO: Starting service SERVICE (2009-06-17 16:40:37,126 main_org.apache.catalina.core.StandardService)
INFO: Starting Servlet Engine: Apache Tomcat/5.5.27 (2009-06-17 16:40:37,133 main_org.apache.catalina.core.StandardEngine)
INFO: Cluster is about to start (2009-06-17 16:40:37,133 main_org.apache.catalina.cluster.tcp.SimpleTcpCluster)
INFO: Start ClusterSender at cluster ENGINE:type=Cluster with name ENGINE:type=ClusterSender (2009-06-17 16:40:37,430 main_org.apache.catalina.cluster.tcp.ReplicationTransmitter)
INFO: Setting cluster mcast soTimeout to 1000 (2009-06-17 16:40:37,440 main_org.apache.catalina.cluster.mcast.McastService)
INFO: Sleeping for 4000 milliseconds to establish cluster membership (2009-06-17 16:40:37,460 main_org.apache.catalina.cluster.mcast.McastService)
INFO: membership mbean registered (ENGINE:type=ClusterMembership) (2009-06-17 16:40:41,478 main_org.apache.catalina.cluster.mcast.McastService)
INFO: XML validation disabled (2009-06-17 16:40:41,504 main_org.apache.catalina.core.StandardHost)
INFO: ContextListener: contextInitialized() (2009-06-17 16:40:43,318 main_org.apache.catalina.core.ContainerBase.[ENGINE].[localhost].[/servlets-examples])
INFO: SessionListener: contextInitialized() (2009-06-17 16:40:43,318 main_org.apache.catalina.core.ContainerBase.[ENGINE].[localhost].[/servlets-examples])
INFO: JvmRouteBinderValve started (2009-06-17 16:40:43,391 main_org.apache.catalina.cluster.session.JvmRouteBinderValve)
INFO: Starting Coyote HTTP/1.1 on http-172.31.3.37-8080 (2009-06-17 16:40:43,467 main_org.apache.coyote.http11.Http11BaseProtocol)
INFO: JK: ajp13 listening on /172.31.3.37:8009 (2009-06-17 16:40:43,640 main_org.apache.jk.common.ChannelSocket)
INFO: Jk running ID=0 time=0/99 config=null (2009-06-17 16:40:43,648 main_org.apache.jk.server.JkMain)
INFO: Find registry server-registry.xml at classpath resource (2009-06-17 16:40:44,411 main_org.apache.catalina.storeconfig.StoreLoader)
INFO: Server startup in 7485 ms (2009-06-17 16:40:44,506 main_org.apache.catalina.startup.Catalina)
INFO: Replication member added:org.apache.catalina.cluster.mcast.McastMember[tcp://172.31.3.34:9015,SAKAINEW,172.31.3.34,9015, alive=11] (2009-06-17 16:40:50,160 Cluster-MembershipReceiver_org.apache.catalina.cluster.tcp.SimpleTcpCluster
)
ERROR: An exception or error occurred in the container during the request processing (2009-06-17 16:43:44,786 http-172.31.3.37-8080-Processor4_org.apache.catalina.connector.CoyoteAdapter)
java.lang.NullPointerException
at org.apache.catalina.cluster.tcp.ReplicationValve.invoke(ReplicationValve.java:348)
at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:174)
at org.apache.coyote.http11.Http11Processor.process(Http11Processor.java:875)
at org.apache.coyote.http11.Http11BaseProtocol$Http11ConnectionHandler.processConnection(Http11BaseProtocol.java:665)
at org.apache.tomcat.util.net.PoolTcpEndpoint.processSocket(PoolTcpEndpoint.java:528)
at org.apache.tomcat.util.net.LeaderFollowerWorkerThread.runIt(LeaderFollowerWorkerThread.java:81)
at org.apache.tomcat.util.threads.ThreadPool$ControlRunnable.run(ThreadPool.java:689)
at java.lang.Thread.run(Thread.java:595)
ERROR: An exception or error occurred in the container during the request processing (2009-06-17 16:43:49,545 http-172.31.3.37-8080-Processor5_org.apache.catalina.connector.CoyoteAdapter)
java.lang.NullPointerException
at org.apache.catalina.cluster.tcp.ReplicationValve.invoke(ReplicationValve.java:348)
at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:174)
at org.apache.coyote.http11.Http11Processor.process(Http11Processor.java:875)
at org.apache.coyote.http11.Http11BaseProtocol$Http11ConnectionHandler.processConnection(Http11BaseProtocol.java:665)
at org.apache.tomcat.util.net.PoolTcpEndpoint.processSocket(PoolTcpEndpoint.java:528)
at org.apache.tomcat.util.net.LeaderFollowerWorkerThread.runIt(LeaderFollowerWorkerThread.java:81)
at org.apache.tomcat.util.threads.ThreadPool$ControlRunnable.run(ThreadPool.java:689)
at java.lang.Thread.run(Thread.java:595)
server.xml
<Server port="7992" shutdown="SHUTDOWN">
<!-- TOMCAT 5 and 5.5-->
<Listener className="org.apache.catalina.core.AprLifecycleListener" SSLEngine="off" />
<Listener className="org.apache.catalina.mbeans.ServerLifecycleListener" />
<Listener className="org.apache.catalina.mbeans.GlobalResourcesLifecycleListener" />
<Listener className="org.apache.catalina.storeconfig.StoreConfigLifecycleListener"/>
<!-- JNDI -->
<GlobalNamingResources>
</GlobalNamingResources>
<Service name="SERVICE">
<!-- HTTP connector -->
<Connector address="${ClusterIP}" port="8080" maxHttpHeaderSize="8192" protocol="HTTP/1.1"
maxThreads="50" minSpareThreads="5" maxSpareThreads="10"
enableLookups="false" redirectPort="8443" acceptCount="100"
connectionTimeout="0" disableUploadTimeout="true" />
<!-- AJP connector -->
<Connector address="${ClusterIP}" port="8009" protocol="AJP/1.3"
connectionTimeout="600000"
enableLookups="false" redirectPort="8443" />
<!-The Engine -->
<Engine name="ENGINE"
defaultHost="localhost">
<Host name="localhost"
appBase="webapps"
unpackWARs="true"
autoDeploy="false"
xmlValidation="false"
xmlNamespaceAware="false">
<Alias>sakai-new.une.edu.au</Alias>
<Alias>espresso.une.edu.au</Alias>
<Alias>latte.une.edu.au</Alias>
</Host>
<Cluster className="org.apache.catalina.cluster.tcp.SimpleTcpCluster"
manager.expireSessionsOnShutdown="false"
doClusterLog="true"
clusterLogName="clusterlog">
<Membership
className="org.apache.catalina.cluster.mcast.McastService"
mcastAddr="228.0.0.4"
mcastClusterDomain="SAKAINEW"
mcastPort="45564"
mcastFrequency="1000"
mcastDropTime="30000"
recoveryCounter="10"
recoveryEnabled="true"
recoverySleepTime="5000" />
<Receiver
className="org.apache.catalina.cluster.tcp.ReplicationListener"
tcpListenAddress="${ClusterIP}"
tcpListenPort="9015"
tcpSelectorTimeout="100"
tcpThreadCount="5"/>
<Sender
className="org.apache.catalina.cluster.tcp.ReplicationTransmitter"
replicationMode="pooled"
autoConnect="true"
maxPoolSocketLimit="10"
keepAliveTimeout="60000"
keepAliveMaxRequestCount="-1"
waitForAck="true" />
<ClusterListener className="org.apache.catalina.cluster.session.ClusterSessionListener" />
<ClusterListener className="org.apache.catalina.cluster.session.JvmRouteSessionIDBinderListener" />
<Valve className="org.apache.catalina.cluster.tcp.ReplicationValve"
filter=".*\.gif;.*\.js;.*\.css;.*\.png;.*\.jpeg;.*\.jpg;.*\.htm;.*\.html;.*\.txt;"
primaryIndicator="true" />
<Valve className="org.apache.catalina.cluster.session.JvmRouteBinderValve"
enabled="true" />
</Cluster>
</Engine>
</Service>
</Server>
David Nillesen
UNIX Systems Administrator
University of New England
+61 2 6773 2112
------ End of Forwarded Message
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://collab.sakaiproject.org/pipermail/sakai-dev/attachments/20090618/014cb6fa/attachment.html
More information about the sakai-dev
mailing list