[Deploying Sakai] "Too many open files" and search indexing

[Colin Tatham]

Thanks Majeed. The errors start for me before the initial indexing has 
completed, and if I restart Tomcat after it's hung, it continues 
indexing and then gets into the same state again before finishing...

Odd thing is that it doesn't actually seem to be opening too many files 
(according to lsof).

Colin

18/11/11 3:15 AM, Majeed:
> We had a problem last weekend where our dev environment kept running
> into a similar "Too many files" error. Tomcat would crash and not be
> listening on 8080 anymore. We are on Solaris and used the pfiles command
> to determine what files a specific pid had opened. Should be able to use
> lsof on linux. Anyhow the main culprit seemed to be files in the
> "indexwork" directory for us. After 3 weeks of running there were over
> 65K files in there. Adding some code to the setenv.sh to delete the
> contents of indexwork dir upon a tomcat restart resolved the issue for
> us. Not to sure of the side effects yet, as we are pretty new to the app.
>
> local.properties:localIndexBase at org.sakaiproject.search.api.JournalSettings=${sakai.home}/indexwork
>
> Good Luck,
> Majeed
>
> --
> Majeed Abu-Qulbain
> Lead Unix Admin - CISS
> Illinois State University
>
>
>
> On 11/17/2011 10:00 AM, Matthew Buckett wrote:
>> On 17 November 2011 12:00, Colin Tatham<colin.tatham at unimelb.edu.au>  wrote:
>>> 14/11/11 10:50 PM, Matthew Buckett:
>>>> On 14 November 2011 06:42, Colin Tatham<colin.tatham at unimelb.edu.au>    wrote:
>>>>> We increased the value set in /etc/security/limits.conf:
>>> ...
>>>>> which seems to have changed the limit:
>>>>> [sakai at crete ~]$ ulimit -n
>>>>> 65535
>>>>>
>>>>> but I still get the errors?
>>>> How are you starting tomcat? as /etc/security/limits.conf only applies
>>>> to process started after a login, if you have an upstart script then
>>>> they won't bet getting picked up.
>>>
>>> I'm still not having much joy with this, any clues in this bit of the stack trace?
>> Attached is a test class. Download it, compile it and run it on a
>> machine to see how many files the JVM can open:
>>
>> javac OpenFiles.java
>> java OpenFiles
>>
>> On my MacBook I get:
>>
>> Too many open files in system
>> Failed to create tmp file. Already got: 10014
>>
>> On my Ubuntu system:
>>
>> Too many open files in system
>> Failed to create tmp file. Already got: 4087
>>
>>
>>
>> _______________________________________________
>> production mailing list
>> production at collab.sakaiproject.org
>> http://collab.sakaiproject.org/mailman/listinfo/production
>>
>> TO UNSUBSCRIBE: send email toproduction-unsubscribe at collab.sakaiproject.org  with a subject of"unsubscribe"
>
>



-- 
Colin Tatham
Educational Technologist | Learning Environments
University Library | The University of Melbourne
http://www.trs.unimelb.edu.au