[Building Sakai] Elastic Search (SRCH-111)

[John Bush]

Right, I think in such a case what we now would be slightly deficient
for that.  Its certainly possible to run embedded nodes as clients
only.  I did add this ticket, SRCH-114, to capture this idea.  But I
think there is a little more to it than just that, the indexing
workers live in Sakai.  If you turn this on with only one node, you
will get a failure when the indexing kicks off.  I might try running
ES standalone in this configuration and see what happens, it might
just work as is, but I imagine there is probably work to do to support
this configuration.

On Sun, Jan 27, 2013 at 2:47 PM, Adrian Fish <adrian.r.fish at gmail.com> wrote:
> Hi John,
>
> I see where you're coming from with the embedded argument. Searches scale
> linearly with users so search nodes scale linearly with app server nodes, so
> why not combine app and search nodes and scale them together. My point about
> using the Rest api in the integration was that if an institution is already
> running an ES cluster they may want to bolt their Sakai onto that. My
> concern was not that you wouldn't be able to access an embedded ES server
> via REST from outside.
>
> Zhen, I've nothing to do with the SOLR integration: it's Colin Hebert's
> work.
>
> Cheers,
> Adrian.
>
>
> On 25 January 2013 16:29, John Bush <john.bush at rsmart.com> wrote:
>>
>> >
>> > If that's the case, then I agree entirely. It seems mad to be forced to
>> > cluster your sakai app servers just to scale your search thing. I'm not
>> > sure
>> > that's what he is saying though. ...Anybody serious about search will
>> > need an external search thing.
>>
>> Unless the use cases for search changes dramatically, I find it hard
>> to imagine a case where you would need to add nodes just to handle
>> search.  Once things are indexed that work load is not really the
>> significant, and really as you pointed out as content is being created
>> and indexed on the fly its not very significant either.  So I disagree
>> that an embedded approach can't just scale with the normal user load
>> the only change being perhaps how many users you can fit on a node or
>> RAM.
>>
>> Maybe pounds work differently that dollars, but at the end of the day
>> this is all about cost.  If I was to go to any sane operations or IT
>> manager and say if you want search to work in Sakai you can add some
>> more RAM to your existing app server nodes (or maybe do nothing), or
>> you can setup a new server and a potentially a new cluster.  Which
>> option do you think they'd take ?  Configuration, server deployment,
>> procurement of the machines, the knowledge around all that stuff all
>> amounts to cost.  So this argument is not solely about what
>> architectures we like more or think might scale better, at the end of
>> the day its about cost.  Personally, I think an embedded approach is
>> more cost effective.  For rSmart which literally has hundreds of Sakai
>> nodes a change in the cost structure of that magnitude is very
>> significant.  I realize for others the situation is different.
>>
>> The idea that search is somehow the bottleneck of the system that
>> warrants a new app node or that search activity is so great that it
>> poses overall risk to the node just isn't consistent with my
>> experience.  If you really wanted to protect users from risk, I'd
>> start with externalizing msgcntr and samigo.
>>
>> > Surely the integration is using the REST
>> > api, not the internal Java one? I think the embedded/external argument
>> > is
>> > moot.
>>
>> The integration uses the internal Java APIs, but that doesn't mean you
>> couldn't conceivably run ES as a separate server.  The code as is
>> doesn't support that yet, but its certainly possible, but not
>> something I was ever planning on personally implementing, but I don't
>> see the usefulness of such a design.  Understand that even when ES is
>> embedded you can access the REST app directly with curl or whatever,
>> this is in fact how I typically work to create queries or do anything
>> administrative.
>>
>> --
>> John Bush
>> 602-490-0470
>
>



-- 
John Bush
602-490-0470