[Deploying Sakai] Sakai is inaccessible - Failed to compete merge....

[David Horwitz]

Hi Leon,

This indicates your local search index on this node is corrupted. To fix
this:

1) Stop the node
2) Delete the files in
localIndexBase at org.sakaiproject.search.api.JournalSettings (defaults to
{tomcat}/sakai/indexwork)
3) Restart


When running in a cluster this will reload the index from shared storage
(sharedJournalBase at org.sakaiproject.search.api.JournalSettings). We
actualy have our start up script do this before starting the node.

To completly clear the index with all nodes stopped (or search disabled):

1) do steps 1 & 2 above
2) delete the contents of the shared index storage
3) clear the metadata from the database:

truncate table search_journal;
truncate table search_node_status;
truncate table search_segments;
truncate table search_transaction;
truncate table searchbuilderitem;
truncate table searchwriterlock;

(mysql syntax oracle may differ)


Hope this helps

D


On 02/22/2011 07:02 AM, Leon Kolchinsky wrote:
> Hello Guys,
>
> We're running 2.5.4 version of Sakai.
> Today we found out that the application is inaccessible and had to
> restart the server.
> On the startup I can see those errors:
>
> INFO: Server startup in 129663 ms (2011-02-22 15:16:02,112
> main_org.apache.catalina.startup.Catalina)
> WARN: Failed to compete merge of 11923 (2011-02-22 15:17:06,180
> Timer-1_org.sakaiproject.search.journal.impl.MergeUpdateOperation)
> org.sakaiproject.search.transaction.api.IndexTransactionException:
> Failed to delete documents
>         at
> org.sakaiproject.search.journal.impl.JournaledFSIndexStorageUpdateTransactionListener.prepare(JournaledFSIndexStorageUpdateTransactionListener.java:161)
>         at
> org.sakaiproject.search.transaction.impl.IndexTransactionImpl.firePrepare(IndexTransactionImpl.java:312)
>         at
> org.sakaiproject.search.transaction.impl.IndexTransactionImpl.prepare(IndexTransactionImpl.java:146)
>         at
> org.sakaiproject.search.journal.impl.MergeUpdateOperation.runOnce(MergeUpdateOperation.java:94)
>         at
> org.sakaiproject.search.journal.impl.IndexManagementTimerTask.run(IndexManagementTimerTask.java:135)
>         at java.util.TimerThread.mainLoop(Timer.java:512)
>         at java.util.TimerThread.run(Timer.java:462)
> Caused by: java.io.IOException:
> /srv/apache-tomcat-5.5.23/sakai/indexwork/index-import/11872 not a
> directory
>         at org.apache.lucene.store.FSDirectory.init(FSDirectory.java:180)
>         at
> org.apache.lucene.store.FSDirectory.getDirectory(FSDirectory.java:141)
>         at
> org.sakaiproject.search.journal.impl.JournaledFSIndexStorage.getIndexReaderInternal(JournaledFSIndexStorage.java:802)
>         at
> org.sakaiproject.search.journal.impl.JournaledFSIndexStorage.getIndexReader(JournaledFSIndexStorage.java:712)
>         at
> org.sakaiproject.search.journal.impl.JournaledFSIndexStorage.getDeletionIndexReader(JournaledFSIndexStorage.java:545)
>         at
> org.sakaiproject.search.journal.impl.JournaledFSIndexStorageUpdateTransactionListener.prepare(JournaledFSIndexStorageUpdateTransactionListener.java:124)
>         ... 6 more
> WARN: Failed to start merge operation  (2011-02-22 15:17:06,205
> Timer-1_org.sakaiproject.search.journal.impl.MergeUpdateOperation)
> org.sakaiproject.search.journal.api.JournalErrorException: Journal is
> stalled at ID 11923
>         at
> org.sakaiproject.search.journal.impl.JournaledFSIndexStorageUpdateTransactionListener.open(JournaledFSIndexStorageUpdateTransactionListener.java:88)
>         at
> org.sakaiproject.search.transaction.impl.IndexTransactionImpl.fireOpen(IndexTransactionImpl.java:359)
>         at
> org.sakaiproject.search.transaction.impl.IndexTransactionImpl.open(IndexTransactionImpl.java:75)
>         at
> org.sakaiproject.search.journal.impl.MergeUpdateManager.openTransaction(MergeUpdateManager.java:48)
>         at
> org.sakaiproject.search.journal.impl.MergeUpdateOperation.runOnce(MergeUpdateOperation.java:92)
>         at
> org.sakaiproject.search.journal.impl.IndexManagementTimerTask.run(IndexManagementTimerTask.java:135)
>         at java.util.TimerThread.mainLoop(Timer.java:512)
>         at java.util.TimerThread.run(Timer.java:462)
> INFO: Local Merge Operation
>  (2011-02-22 15:17:06,205
> Timer-1_org.sakaiproject.search.journal.impl.MergeUpdateOperation)
> WARN: Failed to compete optimize  (2011-02-22 15:17:06,237
> Timer-1_org.sakaiproject.search.optimize.impl.OptimizeIndexOperation)
> org.sakaiproject.search.optimize.api.OptimizedFailedIndexTransactionException:
> Failed to Optimize indexes
>         at
> org.sakaiproject.search.optimize.impl.OptimizeTransactionListenerImpl.prepare(OptimizeTransactionListenerImpl.java:184)
>         at
> org.sakaiproject.search.transaction.impl.IndexTransactionImpl.firePrepare(IndexTransactionImpl.java:312)
>         at
> org.sakaiproject.search.transaction.impl.IndexTransactionImpl.prepare(IndexTransactionImpl.java:146)
>         at
> org.sakaiproject.search.optimize.impl.OptimizeIndexOperation.runOnce(OptimizeIndexOperation.java:90)
>         at
> org.sakaiproject.search.journal.impl.IndexManagementTimerTask.run(IndexManagementTimerTask.java:135)
>         at java.util.TimerThread.mainLoop(Timer.java:512)
>         at java.util.TimerThread.run(Timer.java:462)
> Caused by: java.io.IOException:
> /srv/apache-tomcat-5.5.23/sakai/indexwork/index-import/11872 not a
> directory
>         at org.apache.lucene.store.FSDirectory.init(FSDirectory.java:180)
>         at
> org.apache.lucene.store.FSDirectory.getDirectory(FSDirectory.java:141)
>         at
> org.sakaiproject.search.optimize.impl.OptimizeTransactionListenerImpl.prepare(OptimizeTransactionListenerImpl.java:168)
>         ... 6 more
>
>
> What does it mean?
> Any resolution?
>
> Thanks,
> Leon
>
>
>
>
> _______________________________________________
> production mailing list
> production at collab.sakaiproject.org
> http://collab.sakaiproject.org/mailman/listinfo/production
>
> TO UNSUBSCRIBE: send email to production-unsubscribe at collab.sakaiproject.org with a subject of "unsubscribe"

-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://collab.sakaiproject.org/pipermail/production/attachments/20110222/a3b6834f/attachment.html