[Building Sakai] Unresponsive jvm .. any suggestions?

[Stephen Marquard]

FWIW, we also sometimes get these two errors from the LDAP code, but they’ve never been correlated with any actual problems:

Exception in thread "Thread-8535" java.lang.RuntimeException: Connection.freeWriteSemaphore(4296): semaphore not owned by any thread
        at com.novell.ldap.Connection.freeWriteSemaphore(Unknown Source)
        at com.novell.ldap.Message.putReply(Unknown Source)
        at com.novell.ldap.Connection$ReaderThread.run(Unknown Source)
        at java.lang.Thread.run(Thread.java:745)

Exception in thread "Thread-9061" java.lang.RuntimeException: Connection.freeWriteSemaphore(4559): thread does not own the semaphore, owned by -3
        at com.novell.ldap.Connection.freeWriteSemaphore(Unknown Source)
        at com.novell.ldap.Message.putReply(Unknown Source)
        at com.novell.ldap.Connection$ReaderThread.run(Unknown Source)
        at java.lang.Thread.run(Thread.java:745)

Have you done a thread dump (kill -3 PID) to see what threads are running when the jvm becomes unresponsive? It could be something unrelated to ldap.

Regards
Stephen

---
Stephen Marquard, Learning Technologies Co-ordinator
Centre for Innovation in Learning and Teaching (CILT, formerly CET and HAESDU)
University of Cape Town
http://www.cet.uct.ac.za<http://www.cet.uct.ac.za/>
stephen.marquard at uct.ac.za<mailto:stephen.marquard at uct.ac.za>
Phone: +27-21-650-5037 Cell: +27-83-500-5290

From: sakai-dev-bounces at collab.sakaiproject.org [mailto:sakai-dev-bounces at collab.sakaiproject.org] On Behalf Of Kenwrick Chan
Sent: 09 May 2014 04:31 AM
To: sakai-dev at collab.sakaiproject.org
Subject: Re: [Building Sakai] Unresponsive jvm .. any suggestions?

Folks,
As it turns out ... we still have a "unresponsive jvm".  From the ldap server side of the house we don't see errors so we were looking for where the source is for the openldap portion of the code that the jldap provider uses?  Is it okay to update to a newer version.

We're still not sure how this relates to our jvm in a unresponsive state but right now that's the only marker in our logs that we have to go by.

Any suggestions on how we can solve this or what approaches we can take to try and look for the source of the issue?

Thanks,
Ken



On Tue, May 6, 2014 at 4:59 PM, Kenwrick Chan <kchan at hawaii.edu<mailto:kchan at hawaii.edu>> wrote:
Folks,
We're been running the same sakai codebase with no changes for well over a year.  (2.7.x, we haven't had the resources to migrate yet).  Recently we've run into an issue where an individual front end app server becomes unresponsive.  In the days following we've upped the available memory to the OS and the tomcat app server, but eventually it just fails to respond.

There seems to be a correlation in our catalina.out file to the following error:

Exception in thread "Thread-29287" java.lang.RuntimeException: Connection.freeWriteSemaphore(28847): semaphore not owned by any thread
at com.novell.ldap.Connection.freeWriteSemaphore(Unknown Source)
at com.novell.ldap.Message.putReply(Unknown Source)
at com.novell.ldap.Connection$ReaderThread.run(Unknown Source)
at java.lang.Thread.run(Thread.java:662)

which is pretty much like:
https://jira.sakaiproject.org/browse/SAK-23292

... enough of these errors and the jvm becomes unresponsive.

There are no errors on the ldap side of the house and our network guys say there isn't an issue there impacting access to our ldap servers.  Any suggestions?

-Kenwrick


________________________________
UNIVERSITY OF CAPE TOWN

This e-mail is subject to the UCT ICT policies and e-mail disclaimer published on our website at http://www.uct.ac.za/about/policies/emaildisclaimer/ or obtainable from +27 21 650 9111. This e-mail is intended only for the person(s) to whom it is addressed. If the e-mail has reached you in error, please notify the author. If you are not the intended recipient of the e-mail you may not use, disclose, copy, redirect or print the content. If this e-mail is not related to the business of UCT it is sent by the sender in the sender's individual capacity.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://collab.sakaiproject.org/pipermail/sakai-dev/attachments/20140509/1651e7ba/attachment.html