[sakai2-tcc] Improving the release process

Sat Apr 6 19:56:20 PDT 2013

Does anyone else have any comments or questions? Otherwise I'll work this up into a proposal.

Cheers,
S

Sent from my iPad

On 03/04/2013, at 23:45, Beth Kirschner <bkirschn at umich.edu> wrote:

> I think moving some projects back into the Sakai core within JIRA would really help non-developers report bugs. It's really confusing to those who just use the system to figure out where to report a problem or request. Sometimes in Sakai, sometimes not.
> 
> - Beth
> 
> On Apr 3, 2013, at 8:39 AM, Steve Swinsburg <steve.swinsburg at gmail.com> wrote:
> 
>> Hi D, inline:
>> 
>>>> 
>>>> The way I see it, is that indies require additional release steps in versioning, binary artifact generation and jira project updates. When I did the 2.8.2 and 2.8.3 releases, these were the steps that took the majority of time, at least 80% of it (steps 1-4 of [1])
>>> 1) OK so 1 area we aim to reduce overhead is in jira admin.
>> 
>> Yes, definitely. I think we've already talked about moving some projects back to the core CLE project.
>> 
>>> 2) I'm not sure what you mean by visioning - if this is the 2 stage release:prepare release perform - it is configurable (Jenkins does a great job of this if you've ever done vanila maven releases from it). On the other hand if your talking about the branch, update branch, tag strategy - well that's legacy Sakai and nothing to do with indies.
>> 
>> No, not to do with the actual release itself, more keeping track of the many different versions that the indies have, so that the master pom can be updated correctly, and then any other dependencies are updated as well. Withva better top level pom this could be done ahead of time, all versions would be known and probably the same, so it would just work.
>> 
>>> 3) from m experience the time that release:perfom took was more to do with generating the reporting - again something we could pare down or skip.
>> 
>> I had to get Maurer to run some deploys as he was closer to the box, some tools were taking me upwards of an hour each. It must be doing a lot of transactional stuff on the repo, or is just really slow at doing it. I have a decent connection though.
>> 
>>> 4) The other are of pain was communicating with the Sakai repo via scp - something that is now in the past thanks to sonatype
>> 
>> It will still need to occur since we need to get the artifacts to sonatype, but the push out to central is automatic.
>> 
>>>> 
>>>> Amalgamating them into the core build means we can just push out source tags, and then one binary deploy of all of the artifacts to the maven repo to facilitate individual/contrib tool builds (step 14 of [1])
>>> While on the surface this sounds very desirable it does sound rather like the legacy system in place prior to 2.8 which was in my experience equally slow, complex and understood by too few people (really only Anthony)
>> 
>> It basically is the old system, with some tweaks. And its better documented now and pretty simple - I used Anthony's info for the 2.8.2+ process. If you eliminate the indies part of my guide and have a better top level pom, then all you need to do is make the tag for each tool and push the binaries. Which could be done within an hour I reckon.
>> 
>> cheers,
>> Steve
>> 
>> 
>>> 
>>> D
>>> 
>>>> 
>>>> cheers,
>>>> Steve
>>>> 
>>>> 
>>>> [1] https://confluence.sakaiproject.org/display/~steve.swinsburg/sakai-2.8.2+release
>>>> 
>>>> 
>>>> 
>>>> On 03/04/2013, at 8:18 PM, David Horwitz <David.Horwitz at uct.ac.za> wrote:
>>>> 
>>>>> While I support the goals, what is unclear to me is how undoing the indie process and reverting to a monolithic build is going to simplify the release process. The indiefication was started with this in mind, while I agree we where not as successful as we hoped these where the very issues we where aiming to facilitate with this process. 
>>>>> 
>>>>> D
>>>>> 
>>>>> 
>>>>> 
>>>>> On Tue, 2013-04-02 at 09:20 -0400, Beth Kirschner wrote:
>>>>>> I agree that the experiment with indies has failed their intended goal and doesn't reflect how Sakai is currently maintained. I worry about the effort to once again re-invent the process, but in general I'm supportive. If we have frequent Sakai releases, this could negate the need for those modules that truly do meet the ideals of an indie (e.g. Lesson Builder, Basic LTI, etc.).
>>>>>> 
>>>>>> - Beth
>>>>>> 
>>>>>> On Apr 2, 2013, at 7:13 AM, Steve Swinsburg <
>>>>>> steve.swinsburg at gmail.com
>>>>>>> wrote:
>>>>>> 
>>>>>>> Hi all
>>>>>>> 
>>>>>>> I'd like to have a discussion around improving the release process with respect to having more releases per year and making the release process itself faster. 
>>>>>>> 
>>>>>>> In most cases, the indies have failed in their intended goal. That is, allow off cycle releases. This would have been a perfect process if all modules were individually maintained and actively developed as project teams could release as they desire. Some projects did this well but for most there was no use. They were versioned along with the main CLE code but required an extra couple of steps to release.
>>>>>>> 
>>>>>>> Coupled with a lot of deployers wanting to have the source for patching or some other reason, it just meant more overhead for doing a local build, and the creation of the -all releases and a lot of confusion about where the source code was, how to pull it in and modify it.
>>>>>>> 
>>>>>>> People expect all of the source to be there in a checkout of an open source project, it wasn't and it was kind of odd - "but the modules are binaries in the repository, unzipped at deploy time, etc etc".
>>>>>>> 
>>>>>>> What follow are numbered tasks that I believe need to occur. They come from solid experience in implementing this process for over twenty applications coming together into one final product. They are not yet formal proposals until we discuss. I've just numbered them for clarity.
>>>>>>> 
>>>>>>> Task 1: Scrap all indies, re-version (if appropriate) back in line with the CLE versions, and clean up the externals and poms etc. 
>>>>>>> 
>>>>>>> Task 2: The speed of releases must increase. We cannot have a fix committed the day after a maintenance tag is released, for it to sit there for months until the next big release happens. We should be able to make a minor release quickly. This absolutely must occur if 1 goes ahead.
>>>>>>> 
>>>>>>> Task 3: To facilitate 2, we must have the branch in a constantly releasable state. That will mean no dependency on snapshot artifacts within tools. This needs a bit more thought but is out there for discussion. I believe with a bit of top level pom work we can make this happen. One thing to keep in mind is offline builds. 
>>>>>>> 
>>>>>>> Task 4: The deploy to Maven Central needs to be expanded. It is currently only being done for indies [1] but needs to be done for all artifacts, as per 2.8 and earlier releases (though they go to the Sakai maven repo). This allows contrib projects to build properly being able to find the correct versions of the artifacts they need. 
>>>>>>> 
>>>>>>> Task 5: Remove dependency on Jenkins as a release platform. Unless Jenkins can be setup as a simple server that runs a build and kicks the artifact out, then it is too complex and you might as well use the command line. Logging in to the server, editing scripts [2] or anything else that needs to be done is silly. And if something goes wrong, we need to troubleshoot the jenkins server. A set of clear instructions should be all that is required.
>>>>>>> 
>>>>>>> Task 6: Rationalisation of the base and master poms should occur where necessary. This will also provide clarity for contrib projects which depend on a mix of base and master poms.
>>>>>>> 
>>>>>>> I propose all of the above with a view for implementation in 2.10.
>>>>>>> 
>>>>>>> In discussing, I beg that people refrain from turning this into a resourcing/volunteering issue and discuss the technical issues at hand. Who does what can come afterwards.
>>>>>>> 
>>>>>>> Finally, I make the above comments, suggestions and potential solutions purely because I want to see this process be improved, as a developer, deployer and someone that does the releases. And that is all.
>>>>>>> 
>>>>>>> regards,
>>>>>>> Steve
>>>>>>> 
>>>>>>> [1]
>>>>>> http://central.maven.org/maven2/org/sakaiproject/
>>>>>> 
>>>>>>> [2]
>>>>>> https://confluence.sakaiproject.org/display/REL/Sakai+CLE+release+guide
>>>>>> 
>>>>>>> _______________________________________________
>>>>>>> sakai2-tcc mailing list
>>>>>> sakai2-tcc at collab.sakaiproject.org
>>>>>> 
>>>>>> http://collab.sakaiproject.org/mailman/listinfo/sakai2-tcc
>>>>>> 
>>>>>> 
>>>>>> _______________________________________________
>>>>>> sakai2-tcc mailing list
>>>>>> 
>>>>>> sakai2-tcc at collab.sakaiproject.org
>>>>>> http://collab.sakaiproject.org/mailman/listinfo/sakai2-tcc
>>>>> 
>>>>> 
>>>>> UNIVERSITY OF CAPE TOWN 
>>>>> 
>>>>> This e-mail is subject to the UCT ICT policies and e-mail disclaimer published on our website at http://www.uct.ac.za/about/policies/emaildisclaimer/ or obtainable from +27 21 650 9111. This e-mail is intended only for the person(s) to whom it is addressed. If the e-mail has reached you in error, please notify the author. If you are not the intended recipient of the e-mail you may not use, disclose, copy, redirect or print the content. If this e-mail is not related to the business of UCT it is sent by the sender in the sender's individual capacity.
>>>> 
>>>> 
>>>> _______________________________________________
>>>> sakai2-tcc mailing list
>>>> 
>>>> sakai2-tcc at collab.sakaiproject.org
>>>> http://collab.sakaiproject.org/mailman/listinfo/sakai2-tcc
>> 
>> _______________________________________________
>> sakai2-tcc mailing list
>> sakai2-tcc at collab.sakaiproject.org
>> http://collab.sakaiproject.org/mailman/listinfo/sakai2-tcc
> 