[Building Sakai] FW: Session replication fails on ReplicationValve invocation
Matthew Jones
jonespm at umich.edu
Wed Jun 17 20:08:10 PDT 2009
By default you're not going to have much luck with tomcat based clustering,
most schools just set up a load balancer. Unicon has done quite a bit of
work to get terracotta clustering support into future (2.7) versions of
sakai. It looks like they're also going to be doing some presentations on it
at the Boston conference.
http://jira.sakaiproject.org/jira/browse/SAK-13324
http://www.unicon.net/node/926
<http://www.unicon.net/node/926>
<http://confluence.sakaiproject.org/confluence/display/TERRA/Home?showChildren=false>We
haven't tried this yet as we're still on 2.5, but later this year as we move
toward future versions of Sakai it's something we're also interested in
exploring.
-Matthew
On Wed, Jun 17, 2009 at 8:36 PM, Gerwood Stewart <gstewar8 at une.edu.au>wrote:
> This is possibly more of a Tomcat issue but has anyone faced this issue
> and have they found any work arounds?
>
> Thanks
> Gerwood
>
> ------ Forwarded Message
> *From: *David Nillesen <dnillese at une.edu.au>
> *Date: *Thu, 18 Jun 2009 10:29:55 +1000
> *To: *Gerwood Stewart <gstewar8 at une.edu.au>
> *Subject: *FW: Session replication fails on ReplicationValve invocation
>
>
>
>
> David Nillesen
> UNIX Systems Administrator
> University of New England
> +61 2 6773 2112
>
>
> *From:* David Nillesen
> *Sent:* Thursday, 18 June 2009 10:24 AM
> *To:* 'users at tomcat.apache.org'
> *Subject:* Session replication fails on ReplicationValve invocation
>
> Hi,
> I’m currently trying to establish a cluster of tomcat 5.5.27
> servers to deploy sakai in, so I’m stuck on version 5.5. It is not an option
> to upgrade to tomcat 6 which from all accounts has much better session
> replication.
>
> I have cluster membership up and working via multicast and the two nodes
> add each other as replication targets. When I try to establish a connection
> to the server from my web browser to the default port of 8080 I get this
> neat error in my catalina.out:
>
> ERROR: An exception or error occurred in the container during the request
> processing (2009-06-17 16:43:49,545
> http-172.31.3.37-8080-Processor5_org.apache.catalina.connector.CoyoteAdapter)
> java.lang.NullPointerException
> at
> org.apache.catalina.cluster.tcp.ReplicationValve.invoke(ReplicationValve.java:348)
>
> I’ll put full errors and configs at the bottom of the email. The errors
> occurs regardless of whether I access an empty/nonexistent context or the
> servlet-examples which are currently installed.
>
> The servers are multi homed with 3 interfaces. Eth0 is the
> primary public interface, eth1 is my iSCSI network and eth2 is the backend
> cluster network. All neatly VLANd from each other in seperate subnets.
> Multicast membership works, I can see the packets arriving via tcpdump and
> the membership propagation is occurring.
> I have tried binding the receivers to both public and private interfaces
> but have had no luck. The receivers are up and appear to be functional. I
> can telnet to them and when I enter gibberish into the console it tell me I
> am sending malformed headers.
> By using tcpdump and netstat I can see that neither tomcat instance is
> originating a sender session to connect to the other servers receiver.
>
> I have tried fastsyncqueue and pooled style senders from the documentation.
> Currently configured for pooled as it looked the simpler config to diagnose.
> jvmRoute is configured via –D options passed to java as a lot of our configs
> are done dynamically so as to have a single server.xml across the cluster
> environments. We have a shared disk backend system to enable this.
>
> Any suggestions for diagnosis, help, fixes or otherwise would be very much
> appreciated.
>
> Regards,
> Dave
>
> Log file and startup:
> INFO: The Apache Tomcat Native library which allows optimal performance in
> production environments was not found on the java.library.path:
> /apps/java/jdk1.5.0_19/jre/lib/amd64/server:/apps/java/jdk1.5.0_19/jre/lib/amd64:/apps/java/jdk1.5
> .0_19/jre/../lib/amd64 (2009-06-17 16:40:36,874
> main_org.apache.catalina.core.AprLifecycleListener)
> INFO: Initializing Coyote HTTP/1.1 on http-172.31.3.37-8080 (2009-06-17
> 16:40:37,019 main_org.apache.coyote.http11.Http11BaseProtocol)
> INFO: Initialization processed in 1400 ms (2009-06-17 16:40:37,021
> main_org.apache.catalina.startup.Catalina)
> INFO: Set JAAS app name ENGINE (2009-06-17 16:40:37,114
> main_org.apache.catalina.realm.JAASRealm)
> INFO: Starting service SERVICE (2009-06-17 16:40:37,126
> main_org.apache.catalina.core.StandardService)
> INFO: Starting Servlet Engine: Apache Tomcat/5.5.27 (2009-06-17
> 16:40:37,133 main_org.apache.catalina.core.StandardEngine)
> INFO: Cluster is about to start (2009-06-17 16:40:37,133
> main_org.apache.catalina.cluster.tcp.SimpleTcpCluster)
> INFO: Start ClusterSender at cluster ENGINE:type=Cluster with name
> ENGINE:type=ClusterSender (2009-06-17 16:40:37,430
> main_org.apache.catalina.cluster.tcp.ReplicationTransmitter)
> INFO: Setting cluster mcast soTimeout to 1000 (2009-06-17 16:40:37,440
> main_org.apache.catalina.cluster.mcast.McastService)
> INFO: Sleeping for 4000 milliseconds to establish cluster membership
> (2009-06-17 16:40:37,460
> main_org.apache.catalina.cluster.mcast.McastService)
> INFO: membership mbean registered (ENGINE:type=ClusterMembership)
> (2009-06-17 16:40:41,478
> main_org.apache.catalina.cluster.mcast.McastService)
> INFO: XML validation disabled (2009-06-17 16:40:41,504
> main_org.apache.catalina.core.StandardHost)
> INFO: ContextListener: contextInitialized() (2009-06-17 16:40:43,318
> main_org.apache.catalina.core.ContainerBase.[ENGINE].[localhost].[/servlets-examples])
> INFO: SessionListener: contextInitialized() (2009-06-17 16:40:43,318
> main_org.apache.catalina.core.ContainerBase.[ENGINE].[localhost].[/servlets-examples])
> INFO: JvmRouteBinderValve started (2009-06-17 16:40:43,391
> main_org.apache.catalina.cluster.session.JvmRouteBinderValve)
> INFO: Starting Coyote HTTP/1.1 on http-172.31.3.37-8080 (2009-06-17
> 16:40:43,467 main_org.apache.coyote.http11.Http11BaseProtocol)
> INFO: JK: ajp13 listening on /172.31.3.37:8009 (2009-06-17 16:40:43,640
> main_org.apache.jk.common.ChannelSocket)
> INFO: Jk running ID=0 time=0/99 config=null (2009-06-17 16:40:43,648
> main_org.apache.jk.server.JkMain)
> INFO: Find registry server-registry.xml at classpath resource (2009-06-17
> 16:40:44,411 main_org.apache.catalina.storeconfig.StoreLoader)
> INFO: Server startup in 7485 ms (2009-06-17 16:40:44,506
> main_org.apache.catalina.startup.Catalina)
> INFO: Replication member
> added:org.apache.catalina.cluster.mcast.McastMember[tcp://172.31.3.34:9015,SAKAINEW,172.31.3.34,9015,
> alive=11] (2009-06-17 16:40:50,160
> Cluster-MembershipReceiver_org.apache.catalina.cluster.tcp.SimpleTcpCluster
> )
> ERROR: An exception or error occurred in the container during the request
> processing (2009-06-17 16:43:44,786
> http-172.31.3.37-8080-Processor4_org.apache.catalina.connector.CoyoteAdapter)
> java.lang.NullPointerException
> at
> org.apache.catalina.cluster.tcp.ReplicationValve.invoke(ReplicationValve.java:348)
> at
> org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:174)
> at
> org.apache.coyote.http11.Http11Processor.process(Http11Processor.java:875)
> at
> org.apache.coyote.http11.Http11BaseProtocol$Http11ConnectionHandler.processConnection(Http11BaseProtocol.java:665)
> at
> org.apache.tomcat.util.net.PoolTcpEndpoint.processSocket(PoolTcpEndpoint.java:528)
> at
> org.apache.tomcat.util.net.LeaderFollowerWorkerThread.runIt(LeaderFollowerWorkerThread.java:81)
> at
> org.apache.tomcat.util.threads.ThreadPool$ControlRunnable.run(ThreadPool.java:689)
> at java.lang.Thread.run(Thread.java:595)
> ERROR: An exception or error occurred in the container during the request
> processing (2009-06-17 16:43:49,545
> http-172.31.3.37-8080-Processor5_org.apache.catalina.connector.CoyoteAdapter)
> java.lang.NullPointerException
> at
> org.apache.catalina.cluster.tcp.ReplicationValve.invoke(ReplicationValve.java:348)
> at
> org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:174)
> at
> org.apache.coyote.http11.Http11Processor.process(Http11Processor.java:875)
> at
> org.apache.coyote.http11.Http11BaseProtocol$Http11ConnectionHandler.processConnection(Http11BaseProtocol.java:665)
> at
> org.apache.tomcat.util.net.PoolTcpEndpoint.processSocket(PoolTcpEndpoint.java:528)
> at
> org.apache.tomcat.util.net.LeaderFollowerWorkerThread.runIt(LeaderFollowerWorkerThread.java:81)
> at
> org.apache.tomcat.util.threads.ThreadPool$ControlRunnable.run(ThreadPool.java:689)
> at java.lang.Thread.run(Thread.java:595)
>
> server.xml
>
> <Server port="7992" shutdown="SHUTDOWN">
>
> <!-- TOMCAT 5 and 5.5-->
> <Listener
> className="org.apache.catalina.core.AprLifecycleListener" SSLEngine="off" />
> <Listener
> className="org.apache.catalina.mbeans.ServerLifecycleListener" />
> <Listener
> className="org.apache.catalina.mbeans.GlobalResourcesLifecycleListener" />
> <Listener
> className="org.apache.catalina.storeconfig.StoreConfigLifecycleListener"/>
>
> <!-- JNDI -->
> <GlobalNamingResources>
> </GlobalNamingResources>
>
> <Service name="SERVICE">
>
> <!-- HTTP connector -->
> <Connector address="${ClusterIP}"
> port="8080" maxHttpHeaderSize="8192" protocol="HTTP/1.1"
> maxThreads="50"
> minSpareThreads="5" maxSpareThreads="10"
> enableLookups="false"
> redirectPort="8443" acceptCount="100"
> connectionTimeout="0"
> disableUploadTimeout="true" />
>
> <!-- AJP connector -->
> <Connector address="${ClusterIP}"
> port="8009" protocol="AJP/1.3"
> connectionTimeout="600000"
> enableLookups="false"
> redirectPort="8443" />
>
> <!—The Engine -->
> <Engine name="ENGINE"
> defaultHost="localhost">
> <Host name="localhost"
>
> appBase="webapps"
>
> unpackWARs="true"
>
> autoDeploy="false"
>
> xmlValidation="false"
>
> xmlNamespaceAware="false">
> <Alias>
> sakai-new.une.edu.au</Alias>
> <Alias>
> espresso.une.edu.au</Alias>
> <Alias>
> latte.une.edu.au</Alias>
> </Host>
> <Cluster
> className="org.apache.catalina.cluster.tcp.SimpleTcpCluster"
>
> manager.expireSessionsOnShutdown="false"
>
> doClusterLog="true"
>
> clusterLogName="clusterlog">
> <Membership
>
> className="org.apache.catalina.cluster.mcast.McastService"
>
> mcastAddr="228.0.0.4"
>
> mcastClusterDomain="SAKAINEW"
>
> mcastPort="45564"
>
> mcastFrequency="1000"
>
> mcastDropTime="30000"
>
> recoveryCounter="10"
>
> recoveryEnabled="true"
> recoverySleepTime="5000"
> />
> <Receiver
>
> className="org.apache.catalina.cluster.tcp.ReplicationListener"
>
> tcpListenAddress="${ClusterIP}"
>
> tcpListenPort="9015"
>
> tcpSelectorTimeout="100"
>
> tcpThreadCount="5"/>
>
> <Sender
>
> className="org.apache.catalina.cluster.tcp.ReplicationTransmitter"
>
> replicationMode="pooled"
>
> autoConnect="true"
>
> maxPoolSocketLimit="10"
>
> keepAliveTimeout="60000"
>
> keepAliveMaxRequestCount="-1"
> waitForAck="true"
> />
> <ClusterListener
> className="org.apache.catalina.cluster.session.ClusterSessionListener" />
> <ClusterListener
> className="org.apache.catalina.cluster.session.JvmRouteSessionIDBinderListener"
> />
> <Valve
> className="org.apache.catalina.cluster.tcp.ReplicationValve"
>
> filter=".*\.gif;.*\.js;.*\.css;.*\.png;.*\.jpeg;.*\.jpg;.*\.htm;.*\.html;.*\.txt;"
> primaryIndicator="true"
> />
> <Valve
> className="org.apache.catalina.cluster.session.JvmRouteBinderValve"
> enabled="true"
> />
> </Cluster>
> </Engine>
> </Service>
> </Server>
>
>
> David Nillesen
> UNIX Systems Administrator
> University of New England
> +61 2 6773 2112
>
>
>
> ------ End of Forwarded Message
>
> _______________________________________________
> sakai-dev mailing list
> sakai-dev at collab.sakaiproject.org
> http://collab.sakaiproject.org/mailman/listinfo/sakai-dev
>
> TO UNSUBSCRIBE: send email to
> sakai-dev-unsubscribe at collab.sakaiproject.org with a subject of
> "unsubscribe"
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://collab.sakaiproject.org/pipermail/sakai-dev/attachments/20090617/4cfa8890/attachment.html
More information about the sakai-dev
mailing list