[sakai2-tcc] Improving the release process

Tue Apr 2 10:55:57 PDT 2013

>From my perspective, the iffiest parts of upgrading are the database updates and property changes. Those two are the source of most of our bugs or problems initially with a new upgrade. The whole hybrid-hibernate auto.ddl schema generation paired with manual database update scripts feels very unstable compared to other products I have experience with.

I found myself upgrading an 8-year-old install of MediaWiki to a current version the other day, and I was impressed with how smooth and thorough their schema migration process was as I upgraded through each and every minor release from 1.4 up through 1.20. Every release added some new transitions, but they were all cumulative, so whenever you'd run the upgrade script, it would recheck all the official changes that had happened since whatever past release they supported upgrading from, and ensure that your schema was up to date. Rails has similar functionality it provides to its applications via a custom DSL that specifies the schema and each and every change. Is there anything like that out there for Java/Hibernate projects? Seems like each tool should just maintain in a certain location a history of all schema transitions since some initial state, and then either the app itself or some outside process could read that data and check the database as necessary. How does rSmart handle such cumulative updates?

All of this is to say that if we're going to move to rapid releases, we need some system for avoiding problems that crop up when folks are trying to upgrade multiple versions in one go. Having a solution for automating and validating this sort of thing would be fantastic.

-dave
________________________________________
From: sakai2-tcc-bounces at collab.sakaiproject.org [sakai2-tcc-bounces at collab.sakaiproject.org] On Behalf Of John Bush [john.bush at rsmart.com]
Sent: Tuesday, April 02, 2013 12:34 PM
To: Beth Kirschner
Cc: sakai2-tcc at collab.sakaiproject.org Committee
Subject: Re: [sakai2-tcc] Improving the release process

I agree with Beth and Steve, the indie idea never worked for us, and
we've just worked around it for years.  Re task 2, we are already
releasing every two weeks or so, so I'm neutral on that.  In addition
to needing task 3 for task 2, I worry about quality assurance.  The
only way to really be confident and do it in a resource sane way is to
have more automated testing.   This is deeper than a server just
starting up.

On Tue, Apr 2, 2013 at 6:20 AM, Beth Kirschner <bkirschn at umich.edu> wrote:
> I agree that the experiment with indies has failed their intended goal and doesn't reflect how Sakai is currently maintained. I worry about the effort to once again re-invent the process, but in general I'm supportive. If we have frequent Sakai releases, this could negate the need for those modules that truly do meet the ideals of an indie (e.g. Lesson Builder, Basic LTI, etc.).
>
> - Beth
>
> On Apr 2, 2013, at 7:13 AM, Steve Swinsburg <steve.swinsburg at gmail.com> wrote:
>
>> Hi all
>>
>> I'd like to have a discussion around improving the release process with respect to having more releases per year and making the release process itself faster.
>>
>> In most cases, the indies have failed in their intended goal. That is, allow off cycle releases. This would have been a perfect process if all modules were individually maintained and actively developed as project teams could release as they desire. Some projects did this well but for most there was no use. They were versioned along with the main CLE code but required an extra couple of steps to release.
>>
>> Coupled with a lot of deployers wanting to have the source for patching or some other reason, it just meant more overhead for doing a local build, and the creation of the -all releases and a lot of confusion about where the source code was, how to pull it in and modify it.
>>
>> People expect all of the source to be there in a checkout of an open source project, it wasn't and it was kind of odd - "but the modules are binaries in the repository, unzipped at deploy time, etc etc".
>>
>> What follow are numbered tasks that I believe need to occur. They come from solid experience in implementing this process for over twenty applications coming together into one final product. They are not yet formal proposals until we discuss. I've just numbered them for clarity.
>>
>> Task 1: Scrap all indies, re-version (if appropriate) back in line with the CLE versions, and clean up the externals and poms etc.
>>
>> Task 2: The speed of releases must increase. We cannot have a fix committed the day after a maintenance tag is released, for it to sit there for months until the next big release happens. We should be able to make a minor release quickly. This absolutely must occur if 1 goes ahead.
>>
>> Task 3: To facilitate 2, we must have the branch in a constantly releasable state. That will mean no dependency on snapshot artifacts within tools. This needs a bit more thought but is out there for discussion. I believe with a bit of top level pom work we can make this happen. One thing to keep in mind is offline builds.
>>
>> Task 4: The deploy to Maven Central needs to be expanded. It is currently only being done for indies [1] but needs to be done for all artifacts, as per 2.8 and earlier releases (though they go to the Sakai maven repo). This allows contrib projects to build properly being able to find the correct versions of the artifacts they need.
>>
>> Task 5: Remove dependency on Jenkins as a release platform. Unless Jenkins can be setup as a simple server that runs a build and kicks the artifact out, then it is too complex and you might as well use the command line. Logging in to the server, editing scripts [2] or anything else that needs to be done is silly. And if something goes wrong, we need to troubleshoot the jenkins server. A set of clear instructions should be all that is required.
>>
>> Task 6: Rationalisation of the base and master poms should occur where necessary. This will also provide clarity for contrib projects which depend on a mix of base and master poms.
>>
>> I propose all of the above with a view for implementation in 2.10.
>>
>> In discussing, I beg that people refrain from turning this into a resourcing/volunteering issue and discuss the technical issues at hand. Who does what can come afterwards.
>>
>> Finally, I make the above comments, suggestions and potential solutions purely because I want to see this process be improved, as a developer, deployer and someone that does the releases. And that is all.
>>
>> regards,
>> Steve
>>
>> [1] http://central.maven.org/maven2/org/sakaiproject/
>> [2] https://confluence.sakaiproject.org/display/REL/Sakai+CLE+release+guide
>> _______________________________________________
>> sakai2-tcc mailing list
>> sakai2-tcc at collab.sakaiproject.org
>> http://collab.sakaiproject.org/mailman/listinfo/sakai2-tcc
>
> _______________________________________________
> sakai2-tcc mailing list
> sakai2-tcc at collab.sakaiproject.org
> http://collab.sakaiproject.org/mailman/listinfo/sakai2-tcc

--
John Bush
602-490-0470
_______________________________________________
sakai2-tcc mailing list
sakai2-tcc at collab.sakaiproject.org
http://collab.sakaiproject.org/mailman/listinfo/sakai2-tcc