[Building Sakai] Elastic Search (SRCH-111)

[Adrian Fish]

Hi John,

I see where you're coming from with the embedded argument. Searches scale
linearly with users so search nodes scale linearly with app server nodes,
so why not combine app and search nodes and scale them together. My point
about using the Rest api in the integration was that if an institution is
already running an ES cluster they may want to bolt their Sakai onto that.
My concern was not that you wouldn't be able to access an embedded ES
server via REST from outside.

Zhen, I've nothing to do with the SOLR integration: it's Colin Hebert's
work.

Cheers,
Adrian.


On 25 January 2013 16:29, John Bush <john.bush at rsmart.com> wrote:

> >
> > If that's the case, then I agree entirely. It seems mad to be forced to
> > cluster your sakai app servers just to scale your search thing. I'm not
> sure
> > that's what he is saying though. ...Anybody serious about search will
> need an external search thing.
>
> Unless the use cases for search changes dramatically, I find it hard
> to imagine a case where you would need to add nodes just to handle
> search.  Once things are indexed that work load is not really the
> significant, and really as you pointed out as content is being created
> and indexed on the fly its not very significant either.  So I disagree
> that an embedded approach can't just scale with the normal user load
> the only change being perhaps how many users you can fit on a node or
> RAM.
>
> Maybe pounds work differently that dollars, but at the end of the day
> this is all about cost.  If I was to go to any sane operations or IT
> manager and say if you want search to work in Sakai you can add some
> more RAM to your existing app server nodes (or maybe do nothing), or
> you can setup a new server and a potentially a new cluster.  Which
> option do you think they'd take ?  Configuration, server deployment,
> procurement of the machines, the knowledge around all that stuff all
> amounts to cost.  So this argument is not solely about what
> architectures we like more or think might scale better, at the end of
> the day its about cost.  Personally, I think an embedded approach is
> more cost effective.  For rSmart which literally has hundreds of Sakai
> nodes a change in the cost structure of that magnitude is very
> significant.  I realize for others the situation is different.
>
> The idea that search is somehow the bottleneck of the system that
> warrants a new app node or that search activity is so great that it
> poses overall risk to the node just isn't consistent with my
> experience.  If you really wanted to protect users from risk, I'd
> start with externalizing msgcntr and samigo.
>
> > Surely the integration is using the REST
> > api, not the internal Java one? I think the embedded/external argument is
> > moot.
>
> The integration uses the internal Java APIs, but that doesn't mean you
> couldn't conceivably run ES as a separate server.  The code as is
> doesn't support that yet, but its certainly possible, but not
> something I was ever planning on personally implementing, but I don't
> see the usefulness of such a design.  Understand that even when ES is
> embedded you can access the REST app directly with curl or whatever,
> this is in fact how I typically work to create queries or do anything
> administrative.
>
> --
> John Bush
> 602-490-0470
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://collab.sakaiproject.org/pipermail/sakai-dev/attachments/20130127/724bb0a6/attachment.html