[Deploying Sakai] [Building Sakai] Tomcat process fails to terminate cleanly after adding kaltura integration

Mike De Simone michael.desimone at rsmart.com
Tue Oct 2 09:55:33 PDT 2012


we typically script the shutdown as follows in order to implement a
definitive unattended stop of tomcat:

- call kill -3 on tomcat pid to capture a thread dump just before shutdown
- call shutdown.sh
- wait about 30 seconds to allow anything that can finish normally to do so
(this value depends upon the speed of your server and how much load is on
tomcat at the time) and can vary from about 15-45s in my experience)
- kill -9 to make sure it's stopped.

I don't recommend using the SIGKILL (-9) immediately for production
environments.  You could add another kill -3 after the 30 second wait to
capture information as Aaron suggests.


Thanks,

-------------------------------
Mike DeSimone
Application Operations Manager
*r**Smart* | 602-490-0473


On Tue, Oct 2, 2012 at 8:20 AM, Aaron Zeckoski <azeckoski at unicon.net> wrote:

> We don't have any annotation scanning going on in the kaltura code.
> You might want to try doing the shutdown while still tailing the logs
> and then trying a "kill -3 {pid}" where the pid is the process id
> (from something like ps) of the tomcat server. This will cause a dump
> in the tomcat log of the JVM thread. If you wait a little while
> longer, it will sometimes show other info (because the thing holding
> stuff up might eventually die after some long timeouts).
>
> You want to look for threads in there which are NOT daemons. For
> example, this is a daemon so you can ignore it because the JVM will
> generally terminate daemons without any hesitation:
> "org.azeckoski.reflectutils.refmap.Finalizer" daemon prio=5
> tid=1093f4800 nid=0x10c64a000 in Object.wait() [10c649000]
>    java.lang.Thread.State: WAITING (on object monitor)
> .....
>
> On the other hand, something like this is not a daemon thread:
> "pool-1-thread-1" prio=5 tid=101c54000 nid=0x116a93000 waiting on
> condition [116a92000]
>    java.lang.Thread.State: WAITING (parking)
>         at sun.misc.Unsafe.park(Native Method)
>         - parking to wait for  <7c6fd3ea8> (a
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
>         at
> java.util.concurrent.locks.LockSupport.park(LockSupport.java:156)
>         at
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
>         at java.util.concurrent.DelayQueue.take(DelayQueue.java:160)
>         at
> java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:609)
>         at
> java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:602)
>
>
> Normally the one (or two) non-daemon threads are the culprits that are
> keeping things from shutting down properly.
>
> -AZ
>
>
> On Tue, Oct 2, 2012 at 7:23 AM, Brian Baillargeon <bbailla2 at uwo.ca> wrote:
> > For me, the thread issues in Certification went away when I removed
> > '@resource' annotations and implemented them with ComponentManager. So,
> > I'm not sure if it's an issue with @resource annotations in spring-mvc
> > projects or something else. Anyway, I just grepped unicon's kaltura
> > source and didn't find any @resource annotations, but personally I'd try
> > removing some annotations first and see if the thread stops that way.
> >
> > On 12-10-02 09:19 AM, Mike Jennings wrote:
> >> Tom,
> >>
> >> A few people have had this issue with Tomcat not shutting down properly.
> >>    It appears that some magic combination of tools causes a specific
> >> thread to hang.  Here at UNC when we had the gradebook2 module installed
> >> this issue would happen and then when we removed this tool it
> >> immediately went away.
> >>
> >> You can take a look at this thread here where there was a lot of
> >> discussion on this issue.
> >>
> >>
> http://sakai-project-mail-list-archives.1343168.n2.nabble.com/Building-Sakai-Tomcat-not-shutting-down-properly-td7410135.html
> >>
> >>
> >> Mike
> >>
> >>
> ==============================================================================
> >> Mike Jennings
> >> Teaching and Learning Developer
> >> University of North Carolina at Chapel Hill
> >>
> >> Office: (919) 843-5013
> >> Cell: (919) 698-3746
> >> E-mail: mike_jennings at unc.edu
> >>
> >> On 10/2/2012 8:51 AM, Tom Hall wrote:
> >>> Hi Everyone
> >>>
> >>> We have just incorporated the Kaltura integration module into our 2.8.x
> >>> (source downloaded June 2011 with many mods) development and test
> >>> environments.  Everything appears to work OK except that the <tomcat
> >>> home>/bin/shutdown.sh script no longer stops the tomcat process.  The
> >>> catalina.out log file shows the usual last line of :
> >>>
> >>>     INFO main org.apache.coyote.http11.Http11BaseProtocol - Stopping
> >>> Coyote HTTP/1.1 on http-8080
> >>>
> >>> and there are no errors during the shutdown process.
> >>>
> >>> Has anyone seen anything like this before.
> >>>
> >>> Thanks
> >>>
> >>> Tom
> >>> _______________________________________________
> >>> sakai-dev mailing list
> >>> sakai-dev at collab.sakaiproject.org
> >>> http://collab.sakaiproject.org/mailman/listinfo/sakai-dev
> >>>
> >>> TO UNSUBSCRIBE: send email to
> sakai-dev-unsubscribe at collab.sakaiproject.org with a subject of
> "unsubscribe"
> >>>
> >> _______________________________________________
> >> sakai-dev mailing list
> >> sakai-dev at collab.sakaiproject.org
> >> http://collab.sakaiproject.org/mailman/listinfo/sakai-dev
> >>
> >> TO UNSUBSCRIBE: send email to
> sakai-dev-unsubscribe at collab.sakaiproject.org with a subject of
> "unsubscribe"
> >
> > _______________________________________________
> > sakai-dev mailing list
> > sakai-dev at collab.sakaiproject.org
> > http://collab.sakaiproject.org/mailman/listinfo/sakai-dev
> >
> > TO UNSUBSCRIBE: send email to
> sakai-dev-unsubscribe at collab.sakaiproject.org with a subject of
> "unsubscribe"
>
>
>
> --
> Aaron Zeckoski - Software Architect - http://tinyurl.com/azprofile
> _______________________________________________
> production mailing list
> production at collab.sakaiproject.org
> http://collab.sakaiproject.org/mailman/listinfo/production
>
> TO UNSUBSCRIBE: send email to
> production-unsubscribe at collab.sakaiproject.org with a subject of
> "unsubscribe"
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://collab.sakaiproject.org/pipermail/production/attachments/20121002/57fe6907/attachment-0001.html 


More information about the production mailing list