[Building Sakai] Elastic Search (SRCH-111)

[John Bush]

>
> If that's the case, then I agree entirely. It seems mad to be forced to
> cluster your sakai app servers just to scale your search thing. I'm not sure
> that's what he is saying though. ...Anybody serious about search will need an external search thing.

Unless the use cases for search changes dramatically, I find it hard
to imagine a case where you would need to add nodes just to handle
search.  Once things are indexed that work load is not really the
significant, and really as you pointed out as content is being created
and indexed on the fly its not very significant either.  So I disagree
that an embedded approach can't just scale with the normal user load
the only change being perhaps how many users you can fit on a node or
RAM.

Maybe pounds work differently that dollars, but at the end of the day
this is all about cost.  If I was to go to any sane operations or IT
manager and say if you want search to work in Sakai you can add some
more RAM to your existing app server nodes (or maybe do nothing), or
you can setup a new server and a potentially a new cluster.  Which
option do you think they'd take ?  Configuration, server deployment,
procurement of the machines, the knowledge around all that stuff all
amounts to cost.  So this argument is not solely about what
architectures we like more or think might scale better, at the end of
the day its about cost.  Personally, I think an embedded approach is
more cost effective.  For rSmart which literally has hundreds of Sakai
nodes a change in the cost structure of that magnitude is very
significant.  I realize for others the situation is different.

The idea that search is somehow the bottleneck of the system that
warrants a new app node or that search activity is so great that it
poses overall risk to the node just isn't consistent with my
experience.  If you really wanted to protect users from risk, I'd
start with externalizing msgcntr and samigo.

> Surely the integration is using the REST
> api, not the internal Java one? I think the embedded/external argument is
> moot.

The integration uses the internal Java APIs, but that doesn't mean you
couldn't conceivably run ES as a separate server.  The code as is
doesn't support that yet, but its certainly possible, but not
something I was ever planning on personally implementing, but I don't
see the usefulness of such a design.  Understand that even when ES is
embedded you can access the REST app directly with curl or whatever,
this is in fact how I typically work to create queries or do anything
administrative.

--
John Bush
602-490-0470