[sakai2-tcc] Infrastructure discussion for future meeting?

Wed Mar 13 10:27:06 PDT 2013

Personally, I would except that any major functionality would work (1)
on a basic single server installation and also would work (2) on a
multiple tomcats installations and finally could optionally work (3)
with a separate server (for the really big installations). I don't
think options that don't support at least 1 and 2 are suitable for
core Sakai (as a contrib thing, sure).

-AZ

On Wed, Mar 13, 2013 at 1:23 PM, John Bush <john.bush at rsmart.com> wrote:
> I think its hard to answer that without knowing what people would want
> to use the broker for.  For example, if our posture is that Sakai only
> pushes data into the broker, there is no Sakai code that actually
> consumes anything.  Then folks would who don't have a use case to
> consume this stuff don't need another server.
>
> Now that probably that is a bad assumption, if you are really wanting
> the broker to deal with internode communication between tomcat nodes
> (for cache invalidation or something like that), then clearly there
> would be an issue, but then again only if you require that, maybe if
> you are that big an extra broker server/farm is ok.  So I'm trying to
> sniff out what people expect from this architecture first.
>
> On Wed, Mar 13, 2013 at 10:06 AM, Aaron Zeckoski <azeckoski at unicon.net> wrote:
>>> would be good to get people's reactions to the queue server living
>>> outside of tomcat and being non-Java based technology.
>>
>> So for the schools who don't want to have to setup other servers (akin
>> to using elasticsearch in their tomcat instead of running a SOLR
>> server), what would you implement as the default?
>>
>> -AZ
>>
>>
>>
>> On Wed, Mar 13, 2013 at 1:02 PM, John Bush <john.bush at rsmart.com> wrote:
>>> So regarding the #15.  On one hand we already have that, I mean you
>>> are using java, you can always launch a thread via timer task or
>>> quartz or whatever.  Its not like we are using ruby or php or
>>> something that doesn't have threads and a giant global interpreter
>>> lock.  We do this sort of thing all over that place already.
>>>
>>> On the other hand if you want to use an asynchronous event request to
>>> actually kick off activity in another jvm/process, you would have to
>>> use something like messageservice.  We've been experimenting with this
>>> for while using activemq and pushing sakai's events into it.  This is
>>> actually how we do all our gradebook exports now (gb2).  We push an
>>> export request onto a queue and then an ESB (mule) picks that up and
>>> does all the dirty work.  Its really a great way of keeping all the
>>> local nasty code out of sakai.  And you can upgrade those worker
>>> processes in your ESB outside of Sakai without having to restart, etc.
>>>
>>> I think this type of integration model is really becoming more and
>>> more necessary, especially if we want to keep local code out of sakai.
>>>  It really makes things much more manageable.  Things like analytics,
>>> warehousing, data integration, etc are all obvious use cases for this.
>>>
>>> Now having used activemq for a few years now, I actually have some
>>> concerns.  Its kinda buggy, its cumbersome to configure (especially
>>> for clusters), its too Java centric (although I think it supports amqp
>>> now), and we've got it setup to live inside Sakai.  I think its
>>> important that the queue infrastructure be language independent, a lot
>>> of stuff you are going to want to do with events can be handled by
>>> scripts very well, so python, php, ruby, scala, etc should be able to
>>> consume these things.  A strick JMS protocol is a bad idea these days.
>>>
>>> So for this one, I think a little more understanding of the needs is
>>> actually warranted.  Because if its just for long running processes
>>> and you are ok with them living in the same JVM, I think we have that
>>> already.  But if you are actually wanting to kick off processes
>>> outside the running JVM then we need something else.
>>>
>>> In addition, I think you can make a good argument that you really want
>>> your broker living in a separate process, I don't think you want it
>>> embedded, unless you have some way to restart it without bringing down
>>> the whole Sakai instance.  So I'm not sure what people think about
>>> this architecture, we've always had this approach of running
>>> everything in a bunch of cloned Tomcats + a database.  I'm actually
>>> more interested in looking at something like rabbitmq for this, but I
>>> have no idea how others much react to a departure from Java/JMS.  From
>>> everything I've read (no practical experience yet) rabbitmq feels like
>>> the winner in this space right now in terms of ease of use,
>>> management, and raw speed.
>>>
>>> This is one rSmart would probably jump in a spearhead cause we are
>>> doing things in this space, but before I go out on a limb I think it
>>> would be good to get people's reactions to the queue server living
>>> outside of tomcat and being non-Java based technology.
>>>
>>> On Wed, Mar 13, 2013 at 9:26 AM, Noah Botimer <botimer at umich.edu> wrote:
>>>> +1, nuanced, as usual:
>>>>
>>>> I don't think we have to wait or do any more list curation or even voting. It can go like this pretty easily:
>>>>
>>>> Someone mentions something that they are interested seeing fixed and can either lead or develop on. Others will either join up or say "nah, not so important to me, but X is -- anyone else in for X?".
>>>>
>>>> It's really that simple.
>>>>
>>>> We will either go breadth-first or depth-first depending on the amount of alignment -- but either is just fine so long as people are clear on what they're working on and what's important to them.
>>>>
>>>> Please do not let us find that awful place where we try to tell people "no, don't work on what's most important to you -- work on this thing WE decided on; you're a team player, right?". If there is time and talent just waiting for any old project, hitching that wagon to someone working on a near and dear issue is the fastest and most effective way forward. Standing around, kicking the dirt, waiting for someone to act and point us all in one direction (and subsequently, grudgingly focusing locally) is how we've spent a lot of the past few years.
>>>>
>>>> For a timely example of how this can work, take the Spring / Hibernate upgrades. They've been outstanding for a long time, but are now close to done. They need some more testing (specifically on Oracle) to build up a bit of confidence and a little conversation (by the TCC) to decide the right time to ask all developers to do full rebuilds. The priority was established by people speaking for the issue. Resource was applied by someone who agreed that it was a good initiative. Now comes a bit more coordination and we can call it done. Time to move to the next.
>>>>
>>>>
>>>> I suggest that these two from Dave's list can be tackled immediately with small net investment and would make good headway for operations:
>>>>
>>>>> 5. Lack of support for database failover except through driver indirection.
>>>>
>>>> First step: undo our weird binding to a very old DBCP and allow a real DataSource to be plugged in -- whatever pool or config you need. Second step: pick the Tomcat pool by default and document how to switch if you need something else (hint: should be very standard and basically link to JDK/Tomcat docs). Third step: add some better connection/query/transaction/request metrics that work OOTB so implementors can better see what's going on and decide what to do about it.
>>>>
>>>>> 15. Lack of an asynchronous request system for long-running processes.
>>>>
>>>> First step: retrace the steps done to integrate JMS / ActiveMQ (MessageService?) and decide if that's the right way to go. Second step: implement something basic and very standard. Third step: move some nasty bits like site duplication to the work queue and set up some good docs about how to queue (standard), query for status (standard), and update the user/browser (specific to our portal/notification services).
>>>>
>>>> But anything anyone else is ready to work on would be just fine, too. Refactoring site-manage. Upgrading tools to JSF 2. Reducing session/state in tools. Thinking about a sane runtime configuration / control panel. Those are all long haul jobs but they start with the first commit.
>>>>
>>>> Thanks,
>>>> -Noah
>>>>
>>>> On Mar 13, 2013, at 1:48 AM, John Bush wrote:
>>>>
>>>>> +1 I like this approach.  Its actionable, clear, keeps us focused and
>>>>> moving forward, and stops us from squabbling over things that are too
>>>>> far out to be actionable.
>>>>>
>>>>> On Tue, Mar 12, 2013 at 9:40 PM, Steve Swinsburg
>>>>> <steve.swinsburg at gmail.com> wrote:
>>>>>> +1 sounds good. Lets get a list of issues that are feasible to address in
>>>>>> the timeframe, then we can vote for one.
>>>>>>
>>>>>> cheers,
>>>>>> Steve
>>>>>>
>>>>>>
>>>>>> On Wed, Mar 13, 2013 at 3:30 PM, David Adams <da1 at vt.edu> wrote:
>>>>>>>
>>>>>>> Steve Swinsburg wrote:
>>>>>>>> Dave talked about running a list of issues past management to get
>>>>>>>> their approval for resourcing and funding.
>>>>>>>
>>>>>>> I'm doing a poor job of communicating, so I apologize. But running a
>>>>>>> list of issues past anyone is the opposite of what I think should
>>>>>>> happen. The TCC should pick one problem. Then solve it.
>>>>
>>>
>>>
>>>
>>> --
>>> John Bush
>>> 602-490-0470
>>> _______________________________________________
>>> sakai2-tcc mailing list
>>> sakai2-tcc at collab.sakaiproject.org
>>> http://collab.sakaiproject.org/mailman/listinfo/sakai2-tcc
>>
>>
>>
>> --
>> Aaron Zeckoski - Software Architect - http://tinyurl.com/azprofile
>> _______________________________________________
>> sakai2-tcc mailing list
>> sakai2-tcc at collab.sakaiproject.org
>> http://collab.sakaiproject.org/mailman/listinfo/sakai2-tcc
>
>
>
> --
> John Bush
> 602-490-0470
> _______________________________________________
> sakai2-tcc mailing list
> sakai2-tcc at collab.sakaiproject.org
> http://collab.sakaiproject.org/mailman/listinfo/sakai2-tcc

-- 
Aaron Zeckoski - Software Architect - http://tinyurl.com/azprofile