[Building Sakai] FW: Session replication fails on ReplicationValve invocation
Cris J. Holdorph
holdorph at unicon.net
Thu Jun 18 19:16:14 PDT 2009
Matthew is right, I will be doing two presentations at the Boston Sakai conference on the work Unicon and Wiley did with Terracotta and Sakai. This work was committed to the Sakai Trunk (post 2.6.x, so future 2.7.x). You can learn more about the Terracotta work at this area of confluence.
http://confluence.sakaiproject.org/confluence/display/TERRA/Home
If you're interested in this work I encourage you to attend both presentations. The first one, is crucial to understanding the second.
If you're not able to attend the Sakai conference, here's the really really really really short story.
As stated, Sakai does not use the standard container HTTPSession. Because of the way Sakai components work (these are NOT webapps), you're not able to easily switch Sakai sessions to container based HTTP sessions. So instead, we (Unicon + Wiley) leveraged Terracotta to cluster enable the Sakai Session storage behind the scenes. This does work. There is one limitation. Every tool must be separately cluster enabled. Even if by some miracle you were able to get container HTTP sessions to work, you'd still have to do work in every tool. The fact of life is, to cluster enable tools, you normally must make sure what you put into the session is somehow shareable across the cluster. With HTTP Sessions, this normally means making the objects in the session "Serializable". In Terracotta, it's similar, but without such a hard and fast requirement on the Java Serializable interface.
In the Terracotta Sakai confluence space above, I've given developers a small glimpse at what it takes to Terracotta/cluster enable a tool. This will be expanded on at the Boston conference presentation.
Let me know if you can't make it to the Boston conference and you'd like more information on this work. I'd be happy to arrange a teleconference to discuss it in more detail. I'd love to see some other organizations make use of this work and help begin the process of cluster enabling other tools.
---- Cris J H
----- Original Message -----
From: "Matthew Jones" <jonespm at umich.edu>
To: "Gerwood Stewart" <gstewar8 at une.edu.au>
Cc: "Sakai Dev Mailing List" <sakai-dev at collab.sakaiproject.org>
Sent: Wednesday, June 17, 2009 8:08:10 PM GMT -07:00 U.S. Mountain Time (Arizona)
Subject: Re: [Building Sakai] FW: Session replication fails on ReplicationValve invocation
By default you're not going to have much luck with tomcat based clustering, most schools just set up a load balancer. Unicon has done quite a bit of work to get terracotta clustering support into future (2.7) versions of sakai. It looks like they're also going to be doing some presentations on it at the Boston conference.
http://jira.sakaiproject.org/jira/browse/SAK-13324
http://www.unicon.net/node/926
We haven't tried this yet as we're still on 2.5, but later this year as we move toward future versions of Sakai it's something we're also interested in exploring.
-Matthew
On Wed, Jun 17, 2009 at 8:36 PM, Gerwood Stewart < gstewar8 at une.edu.au > wrote:
This is possibly more of a Tomcat issue but has anyone faced this issue and have they found any work arounds?
Thanks
Gerwood
------ Forwarded Message
From: David Nillesen < dnillese at une.edu.au >
Date: Thu, 18 Jun 2009 10:29:55 +1000
To: Gerwood Stewart < gstewar8 at une.edu.au >
Subject: FW: Session replication fails on ReplicationValve invocation
David Nillesen
UNIX Systems Administrator
University of New England
+61 2 6773 2112
From: David Nillesen
Sent: Thursday, 18 June 2009 10:24 AM
To: ' users at tomcat.apache.org '
Subject: Session replication fails on ReplicationValve invocation
Hi,
I’m currently trying to establish a cluster of tomcat 5.5.27 servers to deploy sakai in, so I’m stuck on version 5.5. It is not an option to upgrade to tomcat 6 which from all accounts has much better session replication.
I have cluster membership up and working via multicast and the two nodes add each other as replication targets. When I try to establish a connection to the server from my web browser to the default port of 8080 I get this neat error in my catalina.out:
ERROR: An exception or error occurred in the container during the request processing (2009-06-17 16:43:49,545 http-172.31.3.37-8080-Processor5_org.apache.catalina.connector.CoyoteAdapter)
java.lang.NullPointerException
at org.apache.catalina.cluster.tcp.ReplicationValve.invoke(ReplicationValve.java:348)
I’ll put full errors and configs at the bottom of the email. The errors occurs regardless of whether I access an empty/nonexistent context or the servlet-examples which are currently installed.
The servers are multi homed with 3 interfaces. Eth0 is the primary public interface, eth1 is my iSCSI network and eth2 is the backend cluster network. All neatly VLANd from each other in seperate subnets. Multicast membership works, I can see the packets arriving via tcpdump and the membership propagation is occurring.
I have tried binding the receivers to both public and private interfaces but have had no luck. The receivers are up and appear to be functional. I can telnet to them and when I enter gibberish into the console it tell me I am sending malformed headers.
By using tcpdump and netstat I can see that neither tomcat instance is originating a sender session to connect to the other servers receiver.
I have tried fastsyncqueue and pooled style senders from the documentation. Currently configured for pooled as it looked the simpler config to diagnose. jvmRoute is configured via –D options passed to java as a lot of our configs are done dynamically so as to have a single server.xml across the cluster environments. We have a shared disk backend system to enable this.
Any suggestions for diagnosis, help, fixes or otherwise would be very much appreciated.
Regards,
Dave
Log file and startup:
INFO: The Apache Tomcat Native library which allows optimal performance in production environments was not found on the java.library.path: /apps/java/jdk1.5.0_19/jre/lib/amd64/server:/apps/java/jdk1.5.0_19/jre/lib/amd64:/apps/java/jdk1.5
.0_19/jre/../lib/amd64 (2009-06-17 16:40:36,874 main_org.apache.catalina.core.AprLifecycleListener)
INFO: Initializing Coyote HTTP/1.1 on http-172.31.3.37-8080 (2009-06-17 16:40:37,019 main_org.apache.coyote.http11.Http11BaseProtocol)
INFO: Initialization processed in 1400 ms (2009-06-17 16:40:37,021 main_org.apache.catalina.startup.Catalina)
INFO: Set JAAS app name ENGINE (2009-06-17 16:40:37,114 main_org.apache.catalina.realm.JAASRealm)
INFO: Starting service SERVICE (2009-06-17 16:40:37,126 main_org.apache.catalina.core.StandardService)
INFO: Starting Servlet Engine: Apache Tomcat/5.5.27 (2009-06-17 16:40:37,133 main_org.apache.catalina.core.StandardEngine)
INFO: Cluster is about to start (2009-06-17 16:40:37,133 main_org.apache.catalina.cluster.tcp.SimpleTcpCluster)
INFO: Start ClusterSender at cluster ENGINE:type=Cluster with name ENGINE:type=ClusterSender (2009-06-17 16:40:37,430 main_org.apache.catalina.cluster.tcp.ReplicationTransmitter)
INFO: Setting cluster mcast soTimeout to 1000 (2009-06-17 16:40:37,440 main_org.apache.catalina.cluster.mcast.McastService)
INFO: Sleeping for 4000 milliseconds to establish cluster membership (2009-06-17 16:40:37,460 main_org.apache.catalina.cluster.mcast.McastService)
INFO: membership mbean registered (ENGINE:type=ClusterMembership) (2009-06-17 16:40:41,478 main_org.apache.catalina.cluster.mcast.McastService)
INFO: XML validation disabled (2009-06-17 16:40:41,504 main_org.apache.catalina.core.StandardHost)
INFO: ContextListener: contextInitialized() (2009-06-17 16:40:43,318 main_org.apache.catalina.core.ContainerBase.[ENGINE].[localhost].[/servlets-examples])
INFO: SessionListener: contextInitialized() (2009-06-17 16:40:43,318 main_org.apache.catalina.core.ContainerBase.[ENGINE].[localhost].[/servlets-examples])
INFO: JvmRouteBinderValve started (2009-06-17 16:40:43,391 main_org.apache.catalina.cluster.session.JvmRouteBinderValve)
INFO: Starting Coyote HTTP/1.1 on http-172.31.3.37-8080 (2009-06-17 16:40:43,467 main_org.apache.coyote.http11.Http11BaseProtocol)
INFO: JK: ajp13 listening on / 172.31.3.37:8009 (2009-06-17 16:40:43,640 main_org.apache.jk.common.ChannelSocket)
INFO: Jk running ID=0 time=0/99 config=null (2009-06-17 16:40:43,648 main_org.apache.jk.server.JkMain)
INFO: Find registry server-registry.xml at classpath resource (2009-06-17 16:40:44,411 main_org.apache.catalina.storeconfig.StoreLoader)
INFO: Server startup in 7485 ms (2009-06-17 16:40:44,506 main_org.apache.catalina.startup.Catalina)
INFO: Replication member added:org.apache.catalina.cluster.mcast.McastMember[tcp:// 172.31.3.34:9015 ,SAKAINEW,172.31.3.34,9015, alive=11] (2009-06-17 16:40:50,160 Cluster-MembershipReceiver_org.apache.catalina.cluster.tcp.SimpleTcpCluster
)
ERROR: An exception or error occurred in the container during the request processing (2009-06-17 16:43:44,786 http-172.31.3.37-8080-Processor4_org.apache.catalina.connector.CoyoteAdapter)
java.lang.NullPointerException
at org.apache.catalina.cluster.tcp.ReplicationValve.invoke(ReplicationValve.java:348)
at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:174)
at org.apache.coyote.http11.Http11Processor.process(Http11Processor.java:875)
at org.apache.coyote.http11.Http11BaseProtocol$Http11ConnectionHandler.processConnection(Http11BaseProtocol.java:665)
at org.apache.tomcat.util.net.PoolTcpEndpoint.processSocket(PoolTcpEndpoint.java:528)
at org.apache.tomcat.util.net.LeaderFollowerWorkerThread.runIt(LeaderFollowerWorkerThread.java:81)
at org.apache.tomcat.util.threads.ThreadPool$ControlRunnable.run(ThreadPool.java:689)
at java.lang.Thread.run(Thread.java:595)
ERROR: An exception or error occurred in the container during the request processing (2009-06-17 16:43:49,545 http-172.31.3.37-8080-Processor5_org.apache.catalina.connector.CoyoteAdapter)
java.lang.NullPointerException
at org.apache.catalina.cluster.tcp.ReplicationValve.invoke(ReplicationValve.java:348)
at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:174)
at org.apache.coyote.http11.Http11Processor.process(Http11Processor.java:875)
at org.apache.coyote.http11.Http11BaseProtocol$Http11ConnectionHandler.processConnection(Http11BaseProtocol.java:665)
at org.apache.tomcat.util.net.PoolTcpEndpoint.processSocket(PoolTcpEndpoint.java:528)
at org.apache.tomcat.util.net.LeaderFollowerWorkerThread.runIt(LeaderFollowerWorkerThread.java:81)
at org.apache.tomcat.util.threads.ThreadPool$ControlRunnable.run(ThreadPool.java:689)
at java.lang.Thread.run(Thread.java:595)
server.xml
<Server port="7992" shutdown="SHUTDOWN">
<!-- TOMCAT 5 and 5.5-->
<Listener className="org.apache.catalina.core.AprLifecycleListener" SSLEngine="off" />
<Listener className="org.apache.catalina.mbeans.ServerLifecycleListener" />
<Listener className="org.apache.catalina.mbeans.GlobalResourcesLifecycleListener" />
<Listener className="org.apache.catalina.storeconfig.StoreConfigLifecycleListener"/>
<!-- JNDI -->
<GlobalNamingResources>
</GlobalNamingResources>
<Service name="SERVICE">
<!-- HTTP connector -->
<Connector address="${ClusterIP}" port="8080" maxHttpHeaderSize="8192" protocol="HTTP/1.1"
maxThreads="50" minSpareThreads="5" maxSpareThreads="10"
enableLookups="false" redirectPort="8443" acceptCount="100"
connectionTimeout="0" disableUploadTimeout="true" />
<!-- AJP connector -->
<Connector address="${ClusterIP}" port="8009" protocol="AJP/1.3"
connectionTimeout="600000"
enableLookups="false" redirectPort="8443" />
<!—The Engine -->
<Engine name="ENGINE"
defaultHost="localhost">
<Host name="localhost"
appBase="webapps"
unpackWARs="true"
autoDeploy="false"
xmlValidation="false"
xmlNamespaceAware="false">
<Alias> sakai-new.une.edu.au </Alias>
<Alias> espresso.une.edu.au </Alias>
<Alias> latte.une.edu.au </Alias>
</Host>
<Cluster className="org.apache.catalina.cluster.tcp.SimpleTcpCluster"
manager.expireSessionsOnShutdown="false"
doClusterLog="true"
clusterLogName="clusterlog">
<Membership
className="org.apache.catalina.cluster.mcast.McastService"
mcastAddr="228.0.0.4"
mcastClusterDomain="SAKAINEW"
mcastPort="45564"
mcastFrequency="1000"
mcastDropTime="30000"
recoveryCounter="10"
recoveryEnabled="true"
recoverySleepTime="5000" />
<Receiver
className="org.apache.catalina.cluster.tcp.ReplicationListener"
tcpListenAddress="${ClusterIP}"
tcpListenPort="9015"
tcpSelectorTimeout="100"
tcpThreadCount="5"/>
<Sender
className="org.apache.catalina.cluster.tcp.ReplicationTransmitter"
replicationMode="pooled"
autoConnect="true"
maxPoolSocketLimit="10"
keepAliveTimeout="60000"
keepAliveMaxRequestCount="-1"
waitForAck="true" />
<ClusterListener className="org.apache.catalina.cluster.session.ClusterSessionListener" />
<ClusterListener className="org.apache.catalina.cluster.session.JvmRouteSessionIDBinderListener" />
<Valve className="org.apache.catalina.cluster.tcp.ReplicationValve"
filter=".*\.gif;.*\.js;.*\.css;.*\.png;.*\.jpeg;.*\.jpg;.*\.htm;.*\.html;.*\.txt;"
primaryIndicator="true" />
<Valve className="org.apache.catalina.cluster.session.JvmRouteBinderValve"
enabled="true" />
</Cluster>
</Engine>
</Service>
</Server>
David Nillesen
UNIX Systems Administrator
University of New England
+61 2 6773 2112
------ End of Forwarded Message
_______________________________________________
sakai-dev mailing list
sakai-dev at collab.sakaiproject.org
http://collab.sakaiproject.org/mailman/listinfo/sakai-dev
TO UNSUBSCRIBE: send email to sakai-dev-unsubscribe at collab.sakaiproject.org with a subject of "unsubscribe"
_______________________________________________
sakai-dev mailing list
sakai-dev at collab.sakaiproject.org
http://collab.sakaiproject.org/mailman/listinfo/sakai-dev
TO UNSUBSCRIBE: send email to sakai-dev-unsubscribe at collab.sakaiproject.org with a subject of "unsubscribe"
More information about the sakai-dev
mailing list