[WG: Sakai QA] Request for inclusion in 2.6.1: SAK-16416

[Anthony Whyte]

I agree with Stephen's view although Michael's hesitation is not  
without foundation (pardon the pun).

IMO, we should be strict up front, so to speak, about what is merged  
into the maintenance branch and only merge fixes when there is sign  
off from QA.  Our maintenance releases could then be generated rather  
quickly:  we decide on a revision point that constitutes the next  
release (2.6.1), we copy to a 2.6.1 release branch, clean up the poms  
(we should be able to leverage the Maven release plugin here), write  
up release notes and cut the release.  In other words, no more alpah/ 
beta/rc tags for maintenance releases (e.g., the Tomcat team has not  
put out an beta tag since 5.5.17).  We perform no additional testing  
on the release branch other than to ensure the artifacts generated  
from it (demo, bin, src) start up in Tomcat.  We then update the 2.6.x  
poms version number to the next SNAPSHOT version (e.g., 2.6.2- 
SNAPSHOT) and recommence the cycle.

But to operationalize this we need an increased commitment from the  
community to support active and ongoing QA of maintenance work.

On Noah's question: I'd like to see project teams empower themselves  
to use the fix version to target a fix for an upcoming maintenance  
release (e.g., 2.6.1, or 2.6.2 or 2.6.3, etc.).  The "contract" here  
is that the fix will be made and QA'd properly before the it goes into  
the maintenance branch and tags are cut.  This implies that there  
exists for project teams a well-under general schedule regarding the  
tagging and releasing and that project teams include QA testers and/or  
work with Pete Peterson to secure testing resources to ensure that  
their work is ready for merging.

If, for example, a non-security, non-blocker/critical fix targeted for  
an upcoming maintenance release is not ready, the release management  
working group could decide after consulting with the project team to  
reset the fix version, incrementing by one, if appropriate, to the  
next maintenance release.  These decisions can be documented in Jira  
and communicated via the QA list or a new release management list.

As with Megan I find the current Jira form overly-complicated.  IMO it  
is a topic that we should discuss in an effort to refine and simplify  
the process with a target of providing a strawman proposal for the  
Boston conference.

Cheers,

Anth


On Jun 5, 2009, at 11:42 AM, Michael Korcuska wrote:

> I think this is a good topic for the project planning meetings at the
> conference. One comment inline.
> On Jun 5, 2009, at 17:37, Stephen Marquard wrote:
>
>> I'm not convinced that the "selective component" approach to
>> maintenance releases is really useful. To me it adds a lot of
>> complexity, and means that the maintenance releases bear no
>> resemblance to the 2-6-x branches that people are running in
>> production. It's confusing when an issue is fixed in the maintenance
>> branch but not in a release which post-dates the fix.
>>
>> I would rather see the testing happen as and when issues are moved
>> into the 2-6-x branch, so that it's safe to release a 2.6.x from the
>> maintenance branch at any time.
>
> I'm not sure this is realistic, actually. Some degree of testing is
> needed on the tag to make sure it is working. I'd be interested in
> opinions as to how little regression testing we can do to still have
> confidence that nothing has broken (or, rather, the risk is very  
> small).
>
>>
>> The only exception to this IMO should be security releases, which
>> are strictly the last .x release plus the security fixes, so that
>> they can be done immediately.
>>
>> Also in general I think we'd be better off putting effort into
>> building more automated tests than trying to get a diffuse pool of
>> volunteers to do more regression testing to a release schedule.
>
> Agreed, but efforts to this end have borne limited fruit thus far.
>
>>
>> Regards
>> Stephen
>>
>>>>> Michael Korcuska <mkorcuska at sakaifoundation.org> 06/05/09 4:19 PM
>>>>>>>>
>> I'm not going to get into the procedures (that's up to the QA group),
>> but let me outline a few things we've discussed in the past:
>>
>> * QA is a (possible the most important) bounding factor on how many
>> and which issues can going into a maintenance release. If we try to
>> include everything that is fixed we create a large QA testing effort.
>> Especially regression testing. This delays maintenance releases.  So
>> there is an important balance between the number of things we include
>> and the speed with which we get maintenance releases out the door.  
>> The
>> more issues fixed, the longer it will take.
>>
>> * It is my view that a maintenance release that introduces  
>> regressions
>> is a very bad thing. So they need to be well tested. Unfortunately  
>> the
>> lack of unit test coverage (or automated testing of any kind) in the
>> 2.x code base makes regression testing a manual process.
>>
>> * To reduce the surface area for testing we (the foundation) has
>> adopted a strategy of targeting a limited number of components for
>> maintenance releases. We test those few well and then cut a release.
>> Hopefully we hop onto the next set of components and can cut a second
>> maintenance release "soon" after.
>>
>> * Security issues change the schedule.  They need to happen quickly
>> and so we may change our plans entirely based on the severity of the
>> security issue and how close we are to cutting a release. And when  
>> one
>> of these issues is involved it will now mean (usually) simultaneous
>> releases on the 2.5 and 2.6 series (at least).
>>
>> Now we haven't been fully successful in executing this approach for a
>> variety of reasons. And it is only a good idea to limit the  
>> components
>> involved if, in fact, we can get maintenance releases out fairly
>> quickly I'm expecting Pete P to get things moving more smoothly.  But
>> right now the community is placing too much reliance on the
>> Foundation. We simply need some help :-).
>>
>> Mostly this means help testing maintenance releases. Those who want  
>> to
>> help test will certainly get a bigger voice in what issues/components
>> are addressed first. Makes sense, yes?
>>
>> Another issue is the work involved in release management. It is
>> increasingly the case that the work of release management has been
>> taken up by Anthony.  We could really use some help with that. I'd
>> love to have someone in the community start to manage the maintenance
>> releases--usually the .0 release is the hardest and once things are
>> set up, the maintenance release process is much less burdensome. Any
>> volunteers to manage the upcoming maintenance releases for the 2.6 or
>> 2.5 series?  Anthony will certainly guide you through the first one
>> and be there to assist.....
>>
>> Best,
>>
>> Michael
>>
>>
>>
>> _______________________________________________
>> sakai-qa mailing list
>> sakai-qa at collab.sakaiproject.org
>> http://collab.sakaiproject.org/mailman/listinfo/sakai-qa
>>
>> TO UNSUBSCRIBE: send email to sakai-qa-unsubscribe at collab.sakaiproject.org
>> with a subject of "unsubscribe"
>
> -- 
> Michael Korcuska
> Executive Director, Sakai Foundation
> mkorcuska at sakaifoundation.org
> phone: +1 510-931-6559
> mobile (US): +1 510-599-2586
> skype: mkorcuska
>
>
>
> _______________________________________________
> sakai-qa mailing list
> sakai-qa at collab.sakaiproject.org
> http://collab.sakaiproject.org/mailman/listinfo/sakai-qa
>
> TO UNSUBSCRIBE: send email to sakai-qa-unsubscribe at collab.sakaiproject.org 
>  with a subject of "unsubscribe"
>
>