[Building Sakai] Unresponsive jvm .. any suggestions?

[Kenwrick Chan]

Thanks Sam.
Yes, it seems like the JVM is running out of memory, but we just configured
JMX on one of app servers last night so we hope to get a definitive sense
of things then.

We don't have that
memory.org.sakaiproject.user.api.UserDirectoryService.callCache
configured in sakai.properties so we'll enable that.

With LDAP debug turned we do see
2014-05-05 09:41:13,178 DEBUG TP-Processor352
edu.amc.sakai.user.JLDAPDirectoryProvider
- getUsers(): [Collection size = 96]

and on the LDAP side of house the query returns a large amounts of
[05/May/2014:09:41:12 -1000] conn=5443345 op=1 RESULT err=0 tag=101
nentries=0 etime=0

Users that no longer exist on the LDAP server (I think they've been
de-provisioned, but are still in a 1000(s)+ user orientation course for new
students).

Could a large volume of this type of transaction generate that error?

-Kenwrick




On Wed, May 7, 2014 at 5:15 AM, Sam Ottenhoff <ottenhoff at longsight.com>wrote:

>
>> We're been running the same sakai codebase with no changes for well over
>> a year.  (2.7.x, we haven't had the resources to migrate yet).  Recently
>> we've run into an issue where an individual front end app server becomes
>> unresponsive.  In the days following we've upped the available memory to
>> the OS and the tomcat app server, but eventually it just fails to respond.
>>
>
> Is the JVM running out of memory?  Are you tracking memory use with JMX or
> VisualVM?
>
>
>> There are no errors on the ldap side of the house and our network guys
>> say there isn't an issue there impacting access to our ldap servers.  Any
>> suggestions?
>>
>>
> The JIRA you referenced seemed to point to some LDAP servers becoming
> unresponsive under heavy load.  Are you sure the LDAP server is responding
> correctly to every request?  Have you tried manual connections using a
> command-line tool like ldapsearch?
>
> Also, a side note: the default UserDirectoryService.callCache in Sakai is
> only five minutes by default.  If your user attributes in LDAP are not
> changing very often, this cache should be raised to be much higher (half a
> day or a full day, even).  Increasing that cache looks like this in
> sakai.properties:
>
>
> memory.org.sakaiproject.user.api.UserDirectoryService.callCache=timeToLiveSeconds=86400,timeToIdleSeconds=0,maxElementsInMemory=50000
>
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://collab.sakaiproject.org/pipermail/sakai-dev/attachments/20140507/2d136550/attachment.html