[Deploying Sakai] Sakai is inaccessible - Failed to compete merge....

[Leon Kolchinsky]

Hello Guys,

We're running 2.5.4 version of Sakai.
Today we found out that the application is inaccessible and had to restart
the server.
On the startup I can see those errors:

INFO: Server startup in 129663 ms (2011-02-22 15:16:02,112
main_org.apache.catalina.startup.Catalina)
WARN: Failed to compete merge of 11923 (2011-02-22 15:17:06,180
Timer-1_org.sakaiproject.search.journal.impl.MergeUpdateOperation)
org.sakaiproject.search.transaction.api.IndexTransactionException: Failed to
delete documents
        at
org.sakaiproject.search.journal.impl.JournaledFSIndexStorageUpdateTransactionListener.prepare(JournaledFSIndexStorageUpdateTransactionListener.java:161)
        at
org.sakaiproject.search.transaction.impl.IndexTransactionImpl.firePrepare(IndexTransactionImpl.java:312)
        at
org.sakaiproject.search.transaction.impl.IndexTransactionImpl.prepare(IndexTransactionImpl.java:146)
        at
org.sakaiproject.search.journal.impl.MergeUpdateOperation.runOnce(MergeUpdateOperation.java:94)
        at
org.sakaiproject.search.journal.impl.IndexManagementTimerTask.run(IndexManagementTimerTask.java:135)
        at java.util.TimerThread.mainLoop(Timer.java:512)
        at java.util.TimerThread.run(Timer.java:462)
Caused by: java.io.IOException:
/srv/apache-tomcat-5.5.23/sakai/indexwork/index-import/11872 not a directory
        at org.apache.lucene.store.FSDirectory.init(FSDirectory.java:180)
        at
org.apache.lucene.store.FSDirectory.getDirectory(FSDirectory.java:141)
        at
org.sakaiproject.search.journal.impl.JournaledFSIndexStorage.getIndexReaderInternal(JournaledFSIndexStorage.java:802)
        at
org.sakaiproject.search.journal.impl.JournaledFSIndexStorage.getIndexReader(JournaledFSIndexStorage.java:712)
        at
org.sakaiproject.search.journal.impl.JournaledFSIndexStorage.getDeletionIndexReader(JournaledFSIndexStorage.java:545)
        at
org.sakaiproject.search.journal.impl.JournaledFSIndexStorageUpdateTransactionListener.prepare(JournaledFSIndexStorageUpdateTransactionListener.java:124)
        ... 6 more
WARN: Failed to start merge operation  (2011-02-22 15:17:06,205
Timer-1_org.sakaiproject.search.journal.impl.MergeUpdateOperation)
org.sakaiproject.search.journal.api.JournalErrorException: Journal is
stalled at ID 11923
        at
org.sakaiproject.search.journal.impl.JournaledFSIndexStorageUpdateTransactionListener.open(JournaledFSIndexStorageUpdateTransactionListener.java:88)
        at
org.sakaiproject.search.transaction.impl.IndexTransactionImpl.fireOpen(IndexTransactionImpl.java:359)
        at
org.sakaiproject.search.transaction.impl.IndexTransactionImpl.open(IndexTransactionImpl.java:75)
        at
org.sakaiproject.search.journal.impl.MergeUpdateManager.openTransaction(MergeUpdateManager.java:48)
        at
org.sakaiproject.search.journal.impl.MergeUpdateOperation.runOnce(MergeUpdateOperation.java:92)
        at
org.sakaiproject.search.journal.impl.IndexManagementTimerTask.run(IndexManagementTimerTask.java:135)
        at java.util.TimerThread.mainLoop(Timer.java:512)
        at java.util.TimerThread.run(Timer.java:462)
INFO: Local Merge Operation
 (2011-02-22 15:17:06,205
Timer-1_org.sakaiproject.search.journal.impl.MergeUpdateOperation)
WARN: Failed to compete optimize  (2011-02-22 15:17:06,237
Timer-1_org.sakaiproject.search.optimize.impl.OptimizeIndexOperation)
org.sakaiproject.search.optimize.api.OptimizedFailedIndexTransactionException:
Failed to Optimize indexes
        at
org.sakaiproject.search.optimize.impl.OptimizeTransactionListenerImpl.prepare(OptimizeTransactionListenerImpl.java:184)
        at
org.sakaiproject.search.transaction.impl.IndexTransactionImpl.firePrepare(IndexTransactionImpl.java:312)
        at
org.sakaiproject.search.transaction.impl.IndexTransactionImpl.prepare(IndexTransactionImpl.java:146)
        at
org.sakaiproject.search.optimize.impl.OptimizeIndexOperation.runOnce(OptimizeIndexOperation.java:90)
        at
org.sakaiproject.search.journal.impl.IndexManagementTimerTask.run(IndexManagementTimerTask.java:135)
        at java.util.TimerThread.mainLoop(Timer.java:512)
        at java.util.TimerThread.run(Timer.java:462)
Caused by: java.io.IOException:
/srv/apache-tomcat-5.5.23/sakai/indexwork/index-import/11872 not a directory
        at org.apache.lucene.store.FSDirectory.init(FSDirectory.java:180)
        at
org.apache.lucene.store.FSDirectory.getDirectory(FSDirectory.java:141)
        at
org.sakaiproject.search.optimize.impl.OptimizeTransactionListenerImpl.prepare(OptimizeTransactionListenerImpl.java:168)
        ... 6 more


What does it mean?
Any resolution?

Thanks,
Leon
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://collab.sakaiproject.org/pipermail/production/attachments/20110222/4a4c2355/attachment.html