[Deploying Sakai] Search shared journal ran out of space

[Matthew Buckett]

Recently our search shared journal ran out of space and we not have
search logging errors:

@400000004fe1db0a04d81acc
org.sakaiproject.search.transaction.api.IndexTransactionException:
Failed to delete documents
@400000004fe1db0a04d81eb4 	at
org.sakaiproject.search.journal.impl.JournaledFSIndexStorageUpdateTransactionListener.prepare(JournaledFSIndexStorageUpdateTransactionListener.java:168)
@400000004fe1db0a04d82684 	at
org.sakaiproject.search.transaction.impl.IndexTransactionImpl.firePrepare(IndexTransactionImpl.java:313)
@400000004fe1db0a04d84d94 	at
org.sakaiproject.search.transaction.impl.IndexTransactionImpl.prepare(IndexTransactionImpl.java:147)
@400000004fe1db0a04d8517c 	at
org.sakaiproject.search.journal.impl.MergeUpdateOperation.runOnce(MergeUpdateOperation.java:94)
@400000004fe1db0a04d85564 	at
org.sakaiproject.search.journal.impl.IndexManagementTimerTask.run(IndexManagementTimerTask.java:137)
@400000004fe1db0a04d8611c 	at java.util.TimerThread.mainLoop(Timer.java:512)
@400000004fe1db0a04d86504 	at java.util.TimerThread.run(Timer.java:462)
@400000004fe1db0a04d868ec Caused by: java.io.EOFException: Unexpected
end of ZLIB input stream
@400000004fe1db0a04d86cd4 	at
java.util.zip.InflaterInputStream.fill(InflaterInputStream.java:223)
@400000004fe1db0a04d897cc 	at
java.util.zip.InflaterInputStream.read(InflaterInputStream.java:141)
@400000004fe1db0a04d89bb4 	at
java.util.zip.ZipInputStream.read(ZipInputStream.java:154)
@400000004fe1db0a04d89f9c 	at
java.io.FilterInputStream.read(FilterInputStream.java:90)
@400000004fe1db0a04d8a384 	at
org.sakaiproject.search.util.FileUtils.unpack(FileUtils.java:287)
@400000004fe1db0a04d8a76c 	at
org.sakaiproject.search.journal.impl.SharedFilesystemJournalStorage.retrieveSavePoint(SharedFilesystemJournalStorage.java:219)
@400000004fe1db0a04d8b324 	at
org.sakaiproject.search.journal.impl.JournaledFSIndexStorageUpdateTransactionListener.prepare(JournaledFSIndexStorageUpdateTransactionListener.java:114)
@400000004fe1db0a04d8baf4 	... 6 more
@400000004fe1db0a04f02af4 2012-06-20 15:15:28,082  WARN IndexManager
org.sakaiproject.search.journal.impl.MergeUpdateOperation - Failed to
start merge operation
@400000004fe1db0a04f032c4
org.sakaiproject.search.journal.api.JournalErrorException: Journal is
stalled at ID 93
@400000004fe1db0a04f036ac 	at
org.sakaiproject.search.journal.impl.JournaledFSIndexStorageUpdateTransactionListener.open(JournaledFSIndexStorageUpdateTransactionListener.java:88)
@400000004fe1db0a04f03e7c 	at
org.sakaiproject.search.transaction.impl.IndexTransactionImpl.fireOpen(IndexTransactionImpl.java:360)
@400000004fe1db0a04f059d4 	at
org.sakaiproject.search.transaction.impl.IndexTransactionImpl.open(IndexTransactionImpl.java:75)
@400000004fe1db0a04f05dbc 	at
org.sakaiproject.search.journal.impl.MergeUpdateManager.openTransaction(MergeUpdateManager.java:48)
@400000004fe1db0a04f061a4 	at
org.sakaiproject.search.journal.impl.MergeUpdateOperation.runOnce(MergeUpdateOperation.java:92)
@400000004fe1db0a04f07144 	at
org.sakaiproject.search.journal.impl.IndexManagementTimerTask.run(IndexManagementTimerTask.java:137)
@400000004fe1db0a04f0752c 	at java.util.TimerThread.mainLoop(Timer.java:512)
@400000004fe1db0a04f07914 	at java.util.TimerThread.run(Timer.java:462)

Is there a sensible way to recover without restarting the service?

-- 
  Matthew Buckett
  VLE Developer, LTG, Oxford University Computing Services