[Building Sakai] 10.0 issues with search

[Matthew Jones]

This is just a heads up and some info for anyone who's already upgraded to
Sakai 10.0. There looks to be some issues with running the new search (with
elastic search) in a cluster configuration with more than one shard.

Most of this is tracked on
https://jira.sakaiproject.org/browse/SRCH-134

But the summary is that whenever search tries to move shards it's getting a
ClassNotFound exception back. I'm supposing this is because of how it sets
up it's worker thread to run the transport and it no longer has access to
the classpath of the elastic search jars in the search component. Stephen
Marquard and I have found that by moving elastic search (and all of it's
dependent jars like lucene) into shared that it works for transport and the
error goes away which confirms this. The only other tool that depends on
jars out of here is help, which I'd upgraded yesterday (SAK-27489
<https://jira.sakaiproject.org/browse/SAK-27489>)

So that brings up some followup issues.
- Is there a way to do this without moving the jars in shared AND without
changing this class in elastic search. I'm guessing not, and changing
elastic search to know about our component classloader of have a custom
trasport seems like it's not super worthwhile.
- We seem to remember at the conference someone else saying that we can't
upgrade past a certain version of elastic search. Maybe it was that we
can't upgrade to 1.x, maybe it was related to this. I don't remember those
details.

Anyway, I plan to fix this (for 10.1) by attaching a patch which will move
all the jars into shared and mark them as provided. (Unless someone has a
better idea in the next week or so)

A temporary workaround at the moment is to either
- switch to the old search
search.service.impl=org.sakaiproject.search.component.service.impl.ConcurrentSearchServiceImpl
search.indexbuilder.impl=org.sakaiproject.search.component.service.impl.SearchIndexBuilderImpl
- or only use 1 shard
elasticsearch.index.number_of_shards=1
elasticsearch.index.number_of_replicas=0

Stephen was saying that the cloud-aws based discovery was no longer working
either and I'd planned to drop this and make it optional anyway since it
probably doesn't matter to many people and was like 10MB. I'm not sure what
or why this wasn't working.

Another issue that was reported is that both indexers are still working and
creating files, even though only one is enabled. This should be fixed.
https://jira.sakaiproject.org/browse/SAK-25607

Thanks!
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://collab.sakaiproject.org/pipermail/sakai-dev/attachments/20140716/e8accec5/attachment.html