[WG: Sakai QA] Branch management & QA strategy

Thu Aug 5 00:50:18 PDT 2010

Superficially, this issue is a finger-pointing exercise. We ended up with SakaiScript broken in 2.7.x. Steve S, the developer, is picking on the branch manager for merging it into 2-7-x without testing it afterwards, and I'm picking on Steve S for having set the 2-7-x Merge status field in the first place, and not explicitly noting the KNL dependency in the description or comments. But it's a small issue easily resolved, so what should have been done is neither here nor there.

The more interesting larger issue is how we treat maintenance branches, and the respective responsibilities of developers, branch managers and QA. For example, are branch managers like developers, or are they QA, or neither?

Some data points:

- We have a shortage of branch managers (not enough people volunteering and actively committing time to branch management activities).

- There's a significant backlog in merges to 2-6-x (currently 76 in Sakai 2.6.x Fixed Issues to be Merged - Verified) and always has been.

- The MT is making progress on fixing issues in trunk, but its work has limited impact for sites running Sakai in production (until they upgrade to the next major release) because many of the fixes don't get merged back, and maintenance releases are infrequent. (There's also a question about prioritization, but that's another topic.)

- Some sites only run tags. Either this is because they lack the capacity to run a maintenance branch, or they don't trust maintenance branches so they run tags + patches.

- Periodically, maintenance branch merges cause regressions (sometimes obvious, sometimes subtle).

- Maintenance releases and maintenance branches get an insignificant amount of QA time applied to them (happy to be proved otherwise if someone has figures).

So there are a number of tradeoffs. If for example, Sakai 2.6.0 has hypothetically 100 bugs, and 50 bugfixes are merged from trunk to 2-6-x, and there's a regression risk of 10% for merges which means that 5 new bugs are introduced, is 2-6-x better off than 2.6.0 ? Numerically yes, as it would have 50 + 5 = 55 bugs rather than 100. However, those 10% regressions are potentially 5 new bugs in previously bug-free areas of code. 

There is a case that having known bugs is better than the risk of continuous exposure to new unknown bugs, and I believe this is largely behind the decision of sites that run tags rather than maintenance branches.

My view on the balance of responsibilities between developers and branch managers is that if we make the bar too high for branch managers, (a) the time cost of merging an issue goes way up, (b) we will have even fewer branch managers. So IMO developers need to have more end-to-end responsibilities for their issues and the likely impact on maintenance branches.

I'm also going to venture an opinion that we should reduce the number of issues that are flagged for merging to maintenance branches, as clearly we are not able to keep up with the existing volume in a meaningful way. For example, we could adopt a new practice that only bugfixes rated Critical or Blocker should get flagged for maintenance branches. 

We should also release maintenance tags more frequently (e.g. every 3 months), and keep the volume of changes low enough that meaningful QA can be done on the tags before release.

Regards
Stephen

>>> "Berg, Alan" <A.M.Berg at uva.nl> 8/5/2010 9:01 AM >>> 
Hi all,

This is an interesting debate that should be discussed further. Both side's make valid points.

I missed most of the debate, but I  support Steve S's view that " all fixes that go into a maintenance branch must be tested by the person doing that work, before they are committed."  QA does not have enough resources to have a different perspective.

At the moment we have a mild explosion of tick boxes in Jira. Perhaps we should add one/two for QA testing requested, code review requested, but that would simply slow down the lifecycle. It is a difficult balance.

Alan B

Alan Berg
QA Director - The Sakai Foundation

Senior Developer / Quality Assurance
Group Education and Research Services
Central Computer Services
University of Amsterdam

http://home.uva.nl/a.m.berg
________________________________________
From: sakai-qa-bounces at collab.sakaiproject.org [sakai-qa-bounces at collab.sakaiproject.org] on behalf of Steve Swinsburg [steve.swinsburg at gmail.com]
Sent: 05 August 2010 08:46
To: Stephen Marquard
Cc: David Horwitz; sakai-qa at collab.sakaiproject.org; Alan Berg
Subject: Re: [WG: Sakai QA] 2.7 webservice broken

I'm not debating that the issue wasn't tested at all, as the comments in the JIRA show that it was (or at least verified).

The issue is that it was committed to the branch without further testing to ensure that the fix was appropriate for that branch. That's the crux of my argument, that all fixes that go into a maintenance branch must be tested by the person doing that work, before they are committed.

cheers,
Steve

On 05/08/2010, at 4:08 PM, Stephen Marquard wrote:

> Maybe this issue is a victim of no two people having the same understanding of JIRA processes.
>
> On this particular issue, the comments indeed have evidence of testing, in the form of "Confirmed fixed in trunk." So whether or not the testing was accurate or not, the branch manager followed the process correctly.
>
> As it seems like an interesting debate, CC'ing the QA list.
>
> Regards
> Stephen
>
>>>> Steve Swinsburg <steve.swinsburg at gmail.com> 8/5/2010 7:57 AM >>>
> I'm alarmed by this and quite strongly disagree.
>
> It's been said time and time again that one of the duties of a branch manager is to test every single thing that goes into that branch. It even says so here: http://confluence.sakaiproject.org/display/REL/Branch+Manager+Guidelines under 'When you have a fix you want to merge'.
>
> Yes, there is a part that says "Verified should be ok to merge but check the comments for evidence... (of) testing", so read on.
>
> So the issue had the merge status set and the branch manager picked it up. That doesn't mean it should be blindly committed without being tested first. Anyone could easily go into JIRA and update a closed issue to add a merge status for the branches. Under this process, a branch manager could come along and merge a fix that breaks things, simply because it wasn't tested. And this might not even be picked up until several tags later.
>
> The branches need to be as stable as a rock, and that means testing everything before it's committed.
>
> I'd be surprised if this position wasn't echoed by the QA team.
>
> thanks,
> Steve
>
>
>
> On 05/08/2010, at 3:42 PM, Stephen Marquard wrote:
>
>> From the issue change log, it appears it was created with a 2-7-x status field set to Merge. As it was then Resolved and later Closed by Sam Ottenhof, the branch manager (Jonathan Cook) in this case acted correctly in terms of process by merging it to 2-7-x.
>>
>> We cannot reasonably expect branch managers to do functional testing on changes that they merge - that's why the process is to merge Closed issues rather than Resolved issues, and to rely on developers to set the 2-6-x and 2-7-x status correctly in the first place.
>>
>> Regards
>> Stephen
>>
>>>>> Steve Swinsburg <steve.swinsburg at gmail.com> 8/5/2010 12:35 AM >>>
>> Hi Dave,
>>
>> This http://jira.sakaiproject.org/browse/SAK-18776 should never have been merged to 2.7.
>>
>> It's also obvious that this wasn't tested before being committed which is worrying. If it was, it would be clear that it didn't compile.  Perhaps we need to send out a reminder that all changes in a branch must be thoroughly tested before being committed?
>>
>> I'll fix this.
>>
>> cheers,
>> S
>>
>>
>>
>> On 04/08/2010, at 11:26 PM, David Horwitz wrote:
>>
>>> Hi Steve,
>>>
>>> It seems compilation of SakaiScript is broken in 2.7. It seems that a number of issues related to:
>>>
>>> http://jira.sakaiproject.org/browse/KNL-428
>>>
>>> Have been merged to 2.7 but KNL-428 is not in the latest 1.1 kernel (it was never flagged for inclusion). It seems we either have to:
>>>
>>> 1) Revert the WS merges
>>> 2) Merge KNL-428 and release a new kernel
>>>
>>>
>>> Advice?
>>>
>>> D
>>
>>
>>
>>
>>
>>
>> ###
>> UNIVERSITY OF CAPE TOWN
>>
>> This e-mail is subject to the UCT ICT policies and e-mail disclaimer published on our website at http://www.uct.ac.za/about/policies/emaildisclaimer/ or obtainable from +27 21 650 4500. This e-mail is intended only for the person(s) to whom it is addressed. If the e-mail has reached you in error, please notify the author. If you are not the intended recipient of the e-mail you may not use, disclose, copy, redirect or print the content. If this e-mail is not related to the business of UCT it is sent by the sender in the sender's individual capacity.
>>
>> ###
>>
>
>
>
>
>
>
> ###
> UNIVERSITY OF CAPE TOWN
>
> This e-mail is subject to the UCT ICT policies and e-mail disclaimer published on our website at http://www.uct.ac.za/about/policies/emaildisclaimer/ or obtainable from +27 21 650 4500. This e-mail is intended only for the person(s) to whom it is addressed. If the e-mail has reached you in error, please notify the author. If you are not the intended recipient of the e-mail you may not use, disclose, copy, redirect or print the content. If this e-mail is not related to the business of UCT it is sent by the sender in the sender's individual capacity.
>
> ###
>

_______________________________________________
sakai-qa mailing list
sakai-qa at collab.sakaiproject.org
http://collab.sakaiproject.org/mailman/listinfo/sakai-qa

TO UNSUBSCRIBE: send email to sakai-qa-unsubscribe at collab.sakaiproject.org with a subject of "unsubscribe"

###
UNIVERSITY OF CAPE TOWN 

This e-mail is subject to the UCT ICT policies and e-mail disclaimer published on our website at http://www.uct.ac.za/about/policies/emaildisclaimer/ or obtainable from +27 21 650 4500. This e-mail is intended only for the person(s) to whom it is addressed. If the e-mail has reached you in error, please notify the author. If you are not the intended recipient of the e-mail you may not use, disclose, copy, redirect or print the content. If this e-mail is not related to the business of UCT it is sent by the sender in the sender's individual capacity.

###