[Building Sakai] [Portfolio] [Management] Deprecating reports and warehouse tools.

[Clay Fenlason]

Thanks, John, for the explanation. Very helpful.

~Clay

On Wed, Mar 10, 2010 at 11:30 PM, John Bush <john.bush at rsmart.com> wrote:
> Having work on a few projects that have attempted to leverage this infrastructure I'll chime in with my experiences/thoughts.
>
> Typically its portfolio use cases, K-12, or complete online instruction use cases that wish to leverage reporting use cases.  The problematic issues in Sakai are that typically people what to report on things across sites, and generally Sakai's API's are always structured to operate within the site context.   While leveraging the api's to do so is in theory possible, in practice the looping necessary is too slow.  It also requires Java development in order to do so.  Generally, for data mining, business intelligence use cases a direct SQL type solution is typical.  That is where Sakai's use of xml meta data is problematic.
>
> The warehouse was created to attempt to solve these issues by pulling the data out of sakai into a denormalized tables that are easier to report against.  The warehouse is a complicated home grown way of producing what people typically use views for.  The warehouse was also intended to shield reports from SQL changes that might occur from release to release.
>
> The reports tool on the other hand was an attempt to create a reporting infrastructure to make reports within Sakai that could have some level of parameterization and sharing capabilities.  Depending on what you want to report on warehousing may or may not be necessary.  That's probably why IU can use reports independently of the warehouse, or they have another warehouse solution.
>
> In the end, running the warehouse from within Sakai has been problematic for a number of reason.  The use of thread local and caching mechanisms within the system don't typically expect complete calls to every bit of data and out of memory issues and such can occur.  The warehousing on a very large system can take several hours if not half a day.   The complexity of the Spring configuration is a big learning curve for anyone attempting to build upon it, and there isn't much documentation to help.
>
> In a nutshell having worked with and built upon these tools, we are looking for other solutions, so I think the product council recommendation makes sense.  When you think about the issue more generally you end up looking at business intelligence and warehousing products: think pentaho, cognios, crystal reports, whatever...
>
> While ultimately it would be nice if more of the xml could move out into the database, I don't suspect anyone to sign up for that sort of work.  What we are looking at is just using views for this.   Both mysql and oracle have ways of parsing the xml directly, if you look at the assignment upgrade sql for 2.6 or 2.6.1 (I forget which one) you can see examples of that.  This alone solves a large part of the warehouse issue, although its a little more tricky in CHS where this stuff is encoded.
>
> I think off the shelf business intelligence products are better at solving the report creation problem.  Possibly with just using webservices to push pdf's or excel docs back into Sakai, that avoids some of the authz complications.  But more rich integrations using basic lti or linktool or whatever are conceivable as well.   It shouldn't take weeks to build reports or expose data in Sakai and in my experience even with a lot of knowledge about the system, that is what reports and the warehouse turned out to be in practice.
>
>
> On Mar 10, 2010, at 3:05 AM, Berg, Alan wrote:
>
>> Hi Dev and production list,
>>
>> I have  recommended to the Product Council and the Maintenance team that the reports and warehouse tools are demoted from Sakai 2.7 as Enterprise core tools and placed back into contrib. Please feel free to comment and make your views known.  The  original email sent is given below
>>
>>
>> > > Hi Product council and Maintenance team members,
>> > >
>> > > the MT are meeting Wednesday to discuss depreciation recommendations for the
>> > > PC as part of the Sakai 2.7 life cycle. This is good as the MT are the
>> > > experts who deal with operational reality on a daily basis.
>> > >
>> > > As Interim QA director I have the professional role to signal issues
>> > > clearly. The Reports tool does not work in the QA tag Sakai 2.7 beta 4 and
>> > > that the warehouse tool also appears to be unsupported. After discussion
>> > > with Noah and Beth I believe that the tools do not have sufficient community
>> > > support and are not being used widely enough to justify continued QA
>> > > effort. I therefore strongly recommend moving the tools from Enterprise core
>> > > status back into contrib before the beta 5 tag is released and thus giving
>> > > QA enough time to verify that removal was safe before 2.7 is tagged.
>>
>>
>>
>> Alan
>>
>> Alan Berg
>> Interim QA Director - The Sakai Foundation
>>
>> Senior Developer / Quality Assurance
>> Group Education and Research Services
>> Central Computer Services
>> University of Amsterdam
>>
>> http://home.uva.nl/a.m.berg
>>
>> _______________________________________________
>> management mailing list
>> management at collab.sakaiproject.org
>> http://collab.sakaiproject.org/mailman/listinfo/management
>>
>> TO UNSUBSCRIBE: send email to management-unsubscribe at collab.sakaiproject.org with a subject of "unsubscribe"
>
> _______________________________________________
> portfolio mailing list
> portfolio at collab.sakaiproject.org
> http://collab.sakaiproject.org/mailman/listinfo/portfolio
>
> TO UNSUBSCRIBE: send email to portfolio-unsubscribe at collab.sakaiproject.org with a subject of "unsubscribe"
>