[sakai2-tcc] Infrastructure discussion for future meeting?

[John Bush]

So regarding the #15.  On one hand we already have that, I mean you
are using java, you can always launch a thread via timer task or
quartz or whatever.  Its not like we are using ruby or php or
something that doesn't have threads and a giant global interpreter
lock.  We do this sort of thing all over that place already.

On the other hand if you want to use an asynchronous event request to
actually kick off activity in another jvm/process, you would have to
use something like messageservice.  We've been experimenting with this
for while using activemq and pushing sakai's events into it.  This is
actually how we do all our gradebook exports now (gb2).  We push an
export request onto a queue and then an ESB (mule) picks that up and
does all the dirty work.  Its really a great way of keeping all the
local nasty code out of sakai.  And you can upgrade those worker
processes in your ESB outside of Sakai without having to restart, etc.

I think this type of integration model is really becoming more and
more necessary, especially if we want to keep local code out of sakai.
 It really makes things much more manageable.  Things like analytics,
warehousing, data integration, etc are all obvious use cases for this.

Now having used activemq for a few years now, I actually have some
concerns.  Its kinda buggy, its cumbersome to configure (especially
for clusters), its too Java centric (although I think it supports amqp
now), and we've got it setup to live inside Sakai.  I think its
important that the queue infrastructure be language independent, a lot
of stuff you are going to want to do with events can be handled by
scripts very well, so python, php, ruby, scala, etc should be able to
consume these things.  A strick JMS protocol is a bad idea these days.

So for this one, I think a little more understanding of the needs is
actually warranted.  Because if its just for long running processes
and you are ok with them living in the same JVM, I think we have that
already.  But if you are actually wanting to kick off processes
outside the running JVM then we need something else.

In addition, I think you can make a good argument that you really want
your broker living in a separate process, I don't think you want it
embedded, unless you have some way to restart it without bringing down
the whole Sakai instance.  So I'm not sure what people think about
this architecture, we've always had this approach of running
everything in a bunch of cloned Tomcats + a database.  I'm actually
more interested in looking at something like rabbitmq for this, but I
have no idea how others much react to a departure from Java/JMS.  From
everything I've read (no practical experience yet) rabbitmq feels like
the winner in this space right now in terms of ease of use,
management, and raw speed.

This is one rSmart would probably jump in a spearhead cause we are
doing things in this space, but before I go out on a limb I think it
would be good to get people's reactions to the queue server living
outside of tomcat and being non-Java based technology.

On Wed, Mar 13, 2013 at 9:26 AM, Noah Botimer <botimer at umich.edu> wrote:
> +1, nuanced, as usual:
>
> I don't think we have to wait or do any more list curation or even voting. It can go like this pretty easily:
>
> Someone mentions something that they are interested seeing fixed and can either lead or develop on. Others will either join up or say "nah, not so important to me, but X is -- anyone else in for X?".
>
> It's really that simple.
>
> We will either go breadth-first or depth-first depending on the amount of alignment -- but either is just fine so long as people are clear on what they're working on and what's important to them.
>
> Please do not let us find that awful place where we try to tell people "no, don't work on what's most important to you -- work on this thing WE decided on; you're a team player, right?". If there is time and talent just waiting for any old project, hitching that wagon to someone working on a near and dear issue is the fastest and most effective way forward. Standing around, kicking the dirt, waiting for someone to act and point us all in one direction (and subsequently, grudgingly focusing locally) is how we've spent a lot of the past few years.
>
> For a timely example of how this can work, take the Spring / Hibernate upgrades. They've been outstanding for a long time, but are now close to done. They need some more testing (specifically on Oracle) to build up a bit of confidence and a little conversation (by the TCC) to decide the right time to ask all developers to do full rebuilds. The priority was established by people speaking for the issue. Resource was applied by someone who agreed that it was a good initiative. Now comes a bit more coordination and we can call it done. Time to move to the next.
>
>
> I suggest that these two from Dave's list can be tackled immediately with small net investment and would make good headway for operations:
>
>> 5. Lack of support for database failover except through driver indirection.
>
> First step: undo our weird binding to a very old DBCP and allow a real DataSource to be plugged in -- whatever pool or config you need. Second step: pick the Tomcat pool by default and document how to switch if you need something else (hint: should be very standard and basically link to JDK/Tomcat docs). Third step: add some better connection/query/transaction/request metrics that work OOTB so implementors can better see what's going on and decide what to do about it.
>
>> 15. Lack of an asynchronous request system for long-running processes.
>
> First step: retrace the steps done to integrate JMS / ActiveMQ (MessageService?) and decide if that's the right way to go. Second step: implement something basic and very standard. Third step: move some nasty bits like site duplication to the work queue and set up some good docs about how to queue (standard), query for status (standard), and update the user/browser (specific to our portal/notification services).
>
> But anything anyone else is ready to work on would be just fine, too. Refactoring site-manage. Upgrading tools to JSF 2. Reducing session/state in tools. Thinking about a sane runtime configuration / control panel. Those are all long haul jobs but they start with the first commit.
>
> Thanks,
> -Noah
>
> On Mar 13, 2013, at 1:48 AM, John Bush wrote:
>
>> +1 I like this approach.  Its actionable, clear, keeps us focused and
>> moving forward, and stops us from squabbling over things that are too
>> far out to be actionable.
>>
>> On Tue, Mar 12, 2013 at 9:40 PM, Steve Swinsburg
>> <steve.swinsburg at gmail.com> wrote:
>>> +1 sounds good. Lets get a list of issues that are feasible to address in
>>> the timeframe, then we can vote for one.
>>>
>>> cheers,
>>> Steve
>>>
>>>
>>> On Wed, Mar 13, 2013 at 3:30 PM, David Adams <da1 at vt.edu> wrote:
>>>>
>>>> Steve Swinsburg wrote:
>>>>> Dave talked about running a list of issues past management to get
>>>>> their approval for resourcing and funding.
>>>>
>>>> I'm doing a poor job of communicating, so I apologize. But running a
>>>> list of issues past anyone is the opposite of what I think should
>>>> happen. The TCC should pick one problem. Then solve it.
>



--
John Bush
602-490-0470