[sakai2-tcc] Improving the release process

Tue Apr 2 11:16:49 PDT 2013

Add it to the important unfunded mandates list! Sakai generally has two
systems for database migrations, the legacy system that generates the
initial data from SQL scripts then doesn't touch it afterward (the
SqlService) and Hibernate which is able to do schema migrations but no data
changes. There are a few others (like FOORM) but that's only used for a few
tables.

We'd talked about putting in some system to handle all of this for doing
complex changes and tracking similar to ruby rake migrate. The best in java
appears to be FlyWayDb (http://flywaydb.org/) but Liquibase is also nice.

It's just another one of those things that's worked well enough that
nobodies put effort into fixing across all the tools. Chuck Hedrick added
some special features for Lessonbuilder because of problems that came up
like this: https://jira.sakaiproject.org/browse/LSNBLDR-198

Having something that takes care of this would be pretty important
for continuous release though and is a big reason for the releases as they
are now. (Actually having real upgrade scripts that we test)

On Tue, Apr 2, 2013 at 1:55 PM, Adams, David <da1 at vt.edu> wrote:

> >From my perspective, the iffiest parts of upgrading are the database
> updates and property changes. Those two are the source of most of our bugs
> or problems initially with a new upgrade. The whole hybrid-hibernate
> auto.ddl schema generation paired with manual database update scripts feels
> very unstable compared to other products I have experience with.
>
> I found myself upgrading an 8-year-old install of MediaWiki to a current
> version the other day, and I was impressed with how smooth and thorough
> their schema migration process was as I upgraded through each and every
> minor release from 1.4 up through 1.20. Every release added some new
> transitions, but they were all cumulative, so whenever you'd run the
> upgrade script, it would recheck all the official changes that had happened
> since whatever past release they supported upgrading from, and ensure that
> your schema was up to date. Rails has similar functionality it provides to
> its applications via a custom DSL that specifies the schema and each and
> every change. Is there anything like that out there for Java/Hibernate
> projects? Seems like each tool should just maintain in a certain location a
> history of all schema transitions since some initial state, and then either
> the app itself or some outside process could read that data and check the
> database as necessary. How does rSmart
>  handle such cumulative updates?
>
> All of this is to say that if we're going to move to rapid releases, we
> need some system for avoiding problems that crop up when folks are trying
> to upgrade multiple versions in one go. Having a solution for automating
> and validating this sort of thing would be fantastic.
>
> -dave
> ________________________________________
> From: sakai2-tcc-bounces at collab.sakaiproject.org [
> sakai2-tcc-bounces at collab.sakaiproject.org] On Behalf Of John Bush [
> john.bush at rsmart.com]
> Sent: Tuesday, April 02, 2013 12:34 PM
> To: Beth Kirschner
> Cc: sakai2-tcc at collab.sakaiproject.org Committee
> Subject: Re: [sakai2-tcc] Improving the release process
>
> I agree with Beth and Steve, the indie idea never worked for us, and
> we've just worked around it for years.  Re task 2, we are already
> releasing every two weeks or so, so I'm neutral on that.  In addition
> to needing task 3 for task 2, I worry about quality assurance.  The
> only way to really be confident and do it in a resource sane way is to
> have more automated testing.   This is deeper than a server just
> starting up.
>
> On Tue, Apr 2, 2013 at 6:20 AM, Beth Kirschner <bkirschn at umich.edu> wrote:
> > I agree that the experiment with indies has failed their intended goal
> and doesn't reflect how Sakai is currently maintained. I worry about the
> effort to once again re-invent the process, but in general I'm supportive.
> If we have frequent Sakai releases, this could negate the need for those
> modules that truly do meet the ideals of an indie (e.g. Lesson Builder,
> Basic LTI, etc.).
> >
> > - Beth
> >
> > On Apr 2, 2013, at 7:13 AM, Steve Swinsburg <steve.swinsburg at gmail.com>
> wrote:
> >
> >> Hi all
> >>
> >> I'd like to have a discussion around improving the release process with
> respect to having more releases per year and making the release process
> itself faster.
> >>
> >> In most cases, the indies have failed in their intended goal. That is,
> allow off cycle releases. This would have been a perfect process if all
> modules were individually maintained and actively developed as project
> teams could release as they desire. Some projects did this well but for
> most there was no use. They were versioned along with the main CLE code but
> required an extra couple of steps to release.
> >>
> >> Coupled with a lot of deployers wanting to have the source for patching
> or some other reason, it just meant more overhead for doing a local build,
> and the creation of the -all releases and a lot of confusion about where
> the source code was, how to pull it in and modify it.
> >>
> >> People expect all of the source to be there in a checkout of an open
> source project, it wasn't and it was kind of odd - "but the modules are
> binaries in the repository, unzipped at deploy time, etc etc".
> >>
> >> What follow are numbered tasks that I believe need to occur. They come
> from solid experience in implementing this process for over twenty
> applications coming together into one final product. They are not yet
> formal proposals until we discuss. I've just numbered them for clarity.
> >>
> >> Task 1: Scrap all indies, re-version (if appropriate) back in line with
> the CLE versions, and clean up the externals and poms etc.
> >>
> >> Task 2: The speed of releases must increase. We cannot have a fix
> committed the day after a maintenance tag is released, for it to sit there
> for months until the next big release happens. We should be able to make a
> minor release quickly. This absolutely must occur if 1 goes ahead.
> >>
> >> Task 3: To facilitate 2, we must have the branch in a constantly
> releasable state. That will mean no dependency on snapshot artifacts within
> tools. This needs a bit more thought but is out there for discussion. I
> believe with a bit of top level pom work we can make this happen. One thing
> to keep in mind is offline builds.
> >>
> >> Task 4: The deploy to Maven Central needs to be expanded. It is
> currently only being done for indies [1] but needs to be done for all
> artifacts, as per 2.8 and earlier releases (though they go to the Sakai
> maven repo). This allows contrib projects to build properly being able to
> find the correct versions of the artifacts they need.
> >>
> >> Task 5: Remove dependency on Jenkins as a release platform. Unless
> Jenkins can be setup as a simple server that runs a build and kicks the
> artifact out, then it is too complex and you might as well use the command
> line. Logging in to the server, editing scripts [2] or anything else that
> needs to be done is silly. And if something goes wrong, we need to
> troubleshoot the jenkins server. A set of clear instructions should be all
> that is required.
> >>
> >> Task 6: Rationalisation of the base and master poms should occur where
> necessary. This will also provide clarity for contrib projects which depend
> on a mix of base and master poms.
> >>
> >> I propose all of the above with a view for implementation in 2.10.
> >>
> >> In discussing, I beg that people refrain from turning this into a
> resourcing/volunteering issue and discuss the technical issues at hand. Who
> does what can come afterwards.
> >>
> >> Finally, I make the above comments, suggestions and potential solutions
> purely because I want to see this process be improved, as a developer,
> deployer and someone that does the releases. And that is all.
> >>
> >> regards,
> >> Steve
> >>
> >> [1] http://central.maven.org/maven2/org/sakaiproject/
> >> [2]
> https://confluence.sakaiproject.org/display/REL/Sakai+CLE+release+guide
> >> _______________________________________________
> >> sakai2-tcc mailing list
> >> sakai2-tcc at collab.sakaiproject.org
> >> http://collab.sakaiproject.org/mailman/listinfo/sakai2-tcc
> >
> > _______________________________________________
> > sakai2-tcc mailing list
> > sakai2-tcc at collab.sakaiproject.org
> > http://collab.sakaiproject.org/mailman/listinfo/sakai2-tcc
>
>
>
> --
> John Bush
> 602-490-0470
> _______________________________________________
> sakai2-tcc mailing list
> sakai2-tcc at collab.sakaiproject.org
> http://collab.sakaiproject.org/mailman/listinfo/sakai2-tcc
> _______________________________________________
> sakai2-tcc mailing list
> sakai2-tcc at collab.sakaiproject.org
> http://collab.sakaiproject.org/mailman/listinfo/sakai2-tcc
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://collab.sakaiproject.org/pipermail/sakai2-tcc/attachments/20130402/8d0ae24a/attachment-0001.html 