[sakai2-tcc] Infrastructure discussion for future meeting?

Wed Mar 13 10:53:41 PDT 2013

> Now regarding Aaron's point (2), are we expecting each tomcat node to
> have its own independent broker in that model or somehow the whole
> queue is shared among all the nodes (that would be required for
> internode communication, which I actually haven't heard anyone mention
> a need for yet), or this a single cluster broker model something we
> could reserve point (3).

Typically activemq is setup to run in each JVM with some broadcast
communication protocol and I think it works out which server is the
current primary with an automatic failover. There might be other
options but I never really explored those.

The main issue with only having it on a single node is (A) how do you
decide which node and (B) if that node goes down... what happens to
the messages (generally they stack up in memory)? That's generally the
problem with any system that only has a single node but communicates
outside the execution space but just stating it as FYI for everyone.

-AZ

On Wed, Mar 13, 2013 at 1:47 PM, John Bush <john.bush at rsmart.com> wrote:
> Ok so what if we were to go along those lines, say rally around AMQP,
> or some open standard.  Then we could do as Noah is suggesting include
> activemq or something embedded that supports it, or allow others to
> turn the embedded piece off, and then host some other AMQP server or
> server infrastructure.  At least this way internal Sakai code would be
> bound to a standard that is more forward thinking then just JMS.
>
> And thanks Sam for the examples, we are really talking about long
> lived user initiated processes that need to execute outside the
> request content.
>
> Now regarding Aaron's point (2), are we expecting each tomcat node to
> have its own independent broker in that model or somehow the whole
> queue is shared among all the nodes (that would be required for
> internode communication, which I actually haven't heard anyone mention
> a need for yet), or this a single cluster broker model something we
> could reserve point (3).
>
> On Wed, Mar 13, 2013 at 10:33 AM, Noah Botimer <botimer at umich.edu> wrote:
>> Perfect. This is the kind of technical expertise and interest that usually stays behind our walls. Sounds like we might have a project lead. :-)
>>
>> I have no personal investment in JMS or ActiveMQ, specifically. I'm actually doing a lot of (J)Ruby these days so RabbitMQ, HornetQ, STOMP, AMQP are on my radar. I'd be more than happy to see an out-of-process, language-inspecific model emerge. There's no reason we couldn't, for example, ship ActiveMQ or HornetQ by default and allow people to set up whatever they need, kind of like HSQLDB is packed and runs by default. Sounds great to me.
>>
>> Thanks,
>> -Noah
>>
>> On Mar 13, 2013, at 1:02 PM, John Bush wrote:
>>
>>> So regarding the #15.  On one hand we already have that, I mean you
>>> are using java, you can always launch a thread via timer task or
>>> quartz or whatever.  Its not like we are using ruby or php or
>>> something that doesn't have threads and a giant global interpreter
>>> lock.  We do this sort of thing all over that place already.
>>>
>>> On the other hand if you want to use an asynchronous event request to
>>> actually kick off activity in another jvm/process, you would have to
>>> use something like messageservice.  We've been experimenting with this
>>> for while using activemq and pushing sakai's events into it.  This is
>>> actually how we do all our gradebook exports now (gb2).  We push an
>>> export request onto a queue and then an ESB (mule) picks that up and
>>> does all the dirty work.  Its really a great way of keeping all the
>>> local nasty code out of sakai.  And you can upgrade those worker
>>> processes in your ESB outside of Sakai without having to restart, etc.
>>>
>>> I think this type of integration model is really becoming more and
>>> more necessary, especially if we want to keep local code out of sakai.
>>> It really makes things much more manageable.  Things like analytics,
>>> warehousing, data integration, etc are all obvious use cases for this.
>>>
>>> Now having used activemq for a few years now, I actually have some
>>> concerns.  Its kinda buggy, its cumbersome to configure (especially
>>> for clusters), its too Java centric (although I think it supports amqp
>>> now), and we've got it setup to live inside Sakai.  I think its
>>> important that the queue infrastructure be language independent, a lot
>>> of stuff you are going to want to do with events can be handled by
>>> scripts very well, so python, php, ruby, scala, etc should be able to
>>> consume these things.  A strick JMS protocol is a bad idea these days.
>>>
>>> So for this one, I think a little more understanding of the needs is
>>> actually warranted.  Because if its just for long running processes
>>> and you are ok with them living in the same JVM, I think we have that
>>> already.  But if you are actually wanting to kick off processes
>>> outside the running JVM then we need something else.
>>>
>>> In addition, I think you can make a good argument that you really want
>>> your broker living in a separate process, I don't think you want it
>>> embedded, unless you have some way to restart it without bringing down
>>> the whole Sakai instance.  So I'm not sure what people think about
>>> this architecture, we've always had this approach of running
>>> everything in a bunch of cloned Tomcats + a database.  I'm actually
>>> more interested in looking at something like rabbitmq for this, but I
>>> have no idea how others much react to a departure from Java/JMS.  From
>>> everything I've read (no practical experience yet) rabbitmq feels like
>>> the winner in this space right now in terms of ease of use,
>>> management, and raw speed.
>>>
>>> This is one rSmart would probably jump in a spearhead cause we are
>>> doing things in this space, but before I go out on a limb I think it
>>> would be good to get people's reactions to the queue server living
>>> outside of tomcat and being non-Java based technology.
>>>
>>> On Wed, Mar 13, 2013 at 9:26 AM, Noah Botimer <botimer at umich.edu> wrote:
>>>> +1, nuanced, as usual:
>>>>
>>>> I don't think we have to wait or do any more list curation or even voting. It can go like this pretty easily:
>>>>
>>>> Someone mentions something that they are interested seeing fixed and can either lead or develop on. Others will either join up or say "nah, not so important to me, but X is -- anyone else in for X?".
>>>>
>>>> It's really that simple.
>>>>
>>>> We will either go breadth-first or depth-first depending on the amount of alignment -- but either is just fine so long as people are clear on what they're working on and what's important to them.
>>>>
>>>> Please do not let us find that awful place where we try to tell people "no, don't work on what's most important to you -- work on this thing WE decided on; you're a team player, right?". If there is time and talent just waiting for any old project, hitching that wagon to someone working on a near and dear issue is the fastest and most effective way forward. Standing around, kicking the dirt, waiting for someone to act and point us all in one direction (and subsequently, grudgingly focusing locally) is how we've spent a lot of the past few years.
>>>>
>>>> For a timely example of how this can work, take the Spring / Hibernate upgrades. They've been outstanding for a long time, but are now close to done. They need some more testing (specifically on Oracle) to build up a bit of confidence and a little conversation (by the TCC) to decide the right time to ask all developers to do full rebuilds. The priority was established by people speaking for the issue. Resource was applied by someone who agreed that it was a good initiative. Now comes a bit more coordination and we can call it done. Time to move to the next.
>>>>
>>>>
>>>> I suggest that these two from Dave's list can be tackled immediately with small net investment and would make good headway for operations:
>>>>
>>>>> 5. Lack of support for database failover except through driver indirection.
>>>>
>>>> First step: undo our weird binding to a very old DBCP and allow a real DataSource to be plugged in -- whatever pool or config you need. Second step: pick the Tomcat pool by default and document how to switch if you need something else (hint: should be very standard and basically link to JDK/Tomcat docs). Third step: add some better connection/query/transaction/request metrics that work OOTB so implementors can better see what's going on and decide what to do about it.
>>>>
>>>>> 15. Lack of an asynchronous request system for long-running processes.
>>>>
>>>> First step: retrace the steps done to integrate JMS / ActiveMQ (MessageService?) and decide if that's the right way to go. Second step: implement something basic and very standard. Third step: move some nasty bits like site duplication to the work queue and set up some good docs about how to queue (standard), query for status (standard), and update the user/browser (specific to our portal/notification services).
>>>>
>>>> But anything anyone else is ready to work on would be just fine, too. Refactoring site-manage. Upgrading tools to JSF 2. Reducing session/state in tools. Thinking about a sane runtime configuration / control panel. Those are all long haul jobs but they start with the first commit.
>>>>
>>>> Thanks,
>>>> -Noah
>>>>
>>>> On Mar 13, 2013, at 1:48 AM, John Bush wrote:
>>>>
>>>>> +1 I like this approach.  Its actionable, clear, keeps us focused and
>>>>> moving forward, and stops us from squabbling over things that are too
>>>>> far out to be actionable.
>>>>>
>>>>> On Tue, Mar 12, 2013 at 9:40 PM, Steve Swinsburg
>>>>> <steve.swinsburg at gmail.com> wrote:
>>>>>> +1 sounds good. Lets get a list of issues that are feasible to address in
>>>>>> the timeframe, then we can vote for one.
>>>>>>
>>>>>> cheers,
>>>>>> Steve
>>>>>>
>>>>>>
>>>>>> On Wed, Mar 13, 2013 at 3:30 PM, David Adams <da1 at vt.edu> wrote:
>>>>>>>
>>>>>>> Steve Swinsburg wrote:
>>>>>>>> Dave talked about running a list of issues past management to get
>>>>>>>> their approval for resourcing and funding.
>>>>>>>
>>>>>>> I'm doing a poor job of communicating, so I apologize. But running a
>>>>>>> list of issues past anyone is the opposite of what I think should
>>>>>>> happen. The TCC should pick one problem. Then solve it.
>>>>
>>>
>>>
>>>
>>> --
>>> John Bush
>>> 602-490-0470
>>
>
>
>
> --
> John Bush
> 602-490-0470
> _______________________________________________
> sakai2-tcc mailing list
> sakai2-tcc at collab.sakaiproject.org
> http://collab.sakaiproject.org/mailman/listinfo/sakai2-tcc

-- 
Aaron Zeckoski - Software Architect - http://tinyurl.com/azprofile