[WG: Sakai QA] Branch management & QA strategy

Anthony Whyte arwhyte at umich.edu
Thu Aug 5 07:03:51 PDT 2010


My observations:

1.  all sides in this discussion should jettison the "personal" and focus on what we need to do to improve both the management of branches and the management of Jira tickets.  Passion, after all, has its limits and the exchange here is among friends seeking a better way.

2.  It appears that the fix was committed to the 2.7.x branch based without first confirming that the merged fix did not break the local 2.7.x webservices build.  This was a mistake.  Branch managers should at an absolute minimum ensure that a merged fix does not break the build before committing it to the target branch (I am also admonishing myself on this) irrespective of whether or not their exists a comment ("Confirmed fixed in trunk") from a trusted contributor as is the case in SAK-18776.

3.  Particularly, in the case of the kernel (but also other indie projects) branch managers must ensure that any dependent fixes have both been committed and are available in the latest binaries, snapshot or otherwise, for both trunk and the target branch.  Developers assigned to tickets should also note such requirements in the ticket description and/or in the comments.

3. Tickets must be linked to all other ticket dependencies in order to provide both a visual cue and a highlighted warning that a branch merge cannot occur in isolation.  If a ticket is dependent on work described in another ticket it needs to be linked.  In the case of SAK-18776 this was not done until after fix was merged to 2.7.x.  In partial mitigation, the description included the comment "Remove duplicated isVisible method now that KNL-428 is in", helpful, but in my view insufficient.  Indeed, the description begs the question "in what" and is not at all helpful.  That said, branch managers should be wary of making assumptions based on such comments and should seek clarification, especially when alerted to ticket dependencies. 

4. Regarding testing.  At a minimum, branch managers should ensure that commits do not break a maintenance branch build, especially one that is post- a *.0 release.  Further testing, of course, would require the provision of test plans, of which none are attached to SAK-18776.  A number of project teams currently manage their own branches (e.g., assignments, msgcntr, osp, profile2, assignments and I assumed, webservices) per my recommendations precisely because it's unrealistic to assume that we have at our disposal a stock of branch managers who possess the knowledge required to handle all parts of Sakai 2x.

There is much more to discuss on this topic (especially regarding the observations/suggestions made by Stephen and Steve) especially regarding the issue of testing.  We should discuss this further and implement improvements to our existing process.

Signing off to call into the RM meeting.

Cheers,

Anthony
 

  







On Aug 5, 2010, at 8:32 AM, Steve Swinsburg wrote:

> Since I've been explicitly singled out, which, I don't appreciate one bit considering my efforts to keep stable branches (see this thread for the background issue '[WG:Sakai] 2.7 webservice broken'), I'll jump in.
> 
> Yes, we have a problem with recruitment for branch management. It's a bit dry and its laborious. The pain is reduced if you are merging fixes to a branch which your own institution has some stake in, but it's probably still considered an afterthought. 
> 
> Since indie projects popped up on the scene, I've been of the view that the project teams could manage their own branches. This is what happens with my own project, Profile2. Sure, it's only gone through one real release cycle, but 82 commits have gone back into that branch [1]. For this approach to work, we need to instil the mindset that a project team should actually care about their own project, produce quality code, not blindly commit, and ultimately take some responsibility to help others out.
> 
> 76 issues waiting to be merged to 2.6 is a considerable number, but its not a monstrous task to address with a few people and a few hours. 
> If you want some stats about how much work this might be:
> * After the July conference there were about 180 outstanding issues for 2.5 [2].
> * Now there are four [3].
> * Over 100 of these were merged to the 2.5.x branch, the others had merge attempts which failed majorly and were reassessed as to their viability, or were checked to see if they were appropriate and if not, closed off. 
> * All security issues were merged and all critical issues were merged, even if it meant manually reworking the fix.
> * This took one person one week.
> 
> Every single commit that went into the 2.5.x branch during that time was personally tested before it was committed, as that is the way that every branch commit happens when I do them. I was of the assumption that merges to the branch actually got tested by others as well, but as I have been informed today, this is not required, which is mind boggling. 
> 
> Sure, testing adds some overhead, a restart might even be required which might take an extra minute or two, but in my view, knowing that particular commit won't break the build for countless others, as well as looking like it fixed the job without introducing any other issues, is well worth it.
> 
> Also, we could lessen the amount of QA required at release time enormously if things were tested as they were committed. Natural usage testing of the branch would uncover any other regressions, QA could pickup the rest. We don't need to have one monolithic cycle whereby a single dedicated team goes in and tries to test everything. Things will be missed. The beauty of having an open source community is that thousands of eyes are looking, not just one group.
> 
> I agree that tags should be put out more frequently. But the QA burden is still far too high if things are being blindly committed to the maintenance branch. In Boston I suggested that tags should be taken directly from the maintenance branch rather than a subset of fixes being taken and put into a release tag. Since this is now the practice that is adopted, the branch MUST be stable. This can only occur if more testing occurs as issues are fixed.
> 
> Back on the subject of getting branch work done:
> I have thought for some time that the maintenance team could complement the branch managers. There are now 20 members in the MT [4], each devoting around three hours per week. Doing the math, thats the capability of almost two weeks work done in one sitting.
> 
> Finally, -1 to reducing the number of issues that are flagged for merges. Jira is not just about the instant, it's about reporting. Keeping that merge flag set will show a user searching for something that the issue that is affecting their installation hasn't actually been fixed for their particular version. Often the affects versions aren't set correctly (ie choosing only 2.6.2 when it actually affects 2.6.1 as well). It also gives us a filter in case someone does get a chunk of time to devote to merging a handful of fixes in.
> 
> I'm not trying to be a hero or win an award, I'm trying to have a stable product which I can use and promote. Having this thrown back in my face is disappointing to say the least.
> 
> Steve
> 
> 
> [1] http://jira.sakaiproject.org/secure/IssueNavigator.jspa?mode=hide&requestId=12484
> [2] http://jira.sakaiproject.org/secure/IssueNavigator.jspa?mode=hide&requestId=12485
> [3]http://jira.sakaiproject.org/secure/IssueNavigator.jspa?mode=hide&requestId=11806
> [4] http://confluence.sakaiproject.org/display/MNT/members
> 
> 
> On 05/08/2010, at 7:14 PM, John Norman wrote:
> 
>> FWIW I am largely in agreement with Stephen's suggestions below. I recognise that we are contributing little in this area, so my voice doesn't count for much.
>> John
>> 
>> On 5 Aug 2010, at 08:50, Stephen Marquard wrote:
>> 
>>> Superficially, this issue is a finger-pointing exercise. We ended up with SakaiScript broken in 2.7.x. Steve S, the developer, is picking on the branch manager for merging it into 2-7-x without testing it afterwards, and I'm picking on Steve S for having set the 2-7-x Merge status field in the first place, and not explicitly noting the KNL dependency in the description or comments. But it's a small issue easily resolved, so what should have been done is neither here nor there.
>>> 
>>> The more interesting larger issue is how we treat maintenance branches, and the respective responsibilities of developers, branch managers and QA. For example, are branch managers like developers, or are they QA, or neither?
>>> 
>>> Some data points:
>>> 
>>> - We have a shortage of branch managers (not enough people volunteering and actively committing time to branch management activities).
>>> 
>>> - There's a significant backlog in merges to 2-6-x (currently 76 in Sakai 2.6.x Fixed Issues to be Merged - Verified) and always has been.
>>> 
>>> - The MT is making progress on fixing issues in trunk, but its work has limited impact for sites running Sakai in production (until they upgrade to the next major release) because many of the fixes don't get merged back, and maintenance releases are infrequent. (There's also a question about prioritization, but that's another topic.)
>>> 
>>> - Some sites only run tags. Either this is because they lack the capacity to run a maintenance branch, or they don't trust maintenance branches so they run tags + patches.
>>> 
>>> - Periodically, maintenance branch merges cause regressions (sometimes obvious, sometimes subtle).
>>> 
>>> - Maintenance releases and maintenance branches get an insignificant amount of QA time applied to them (happy to be proved otherwise if someone has figures).
>>> 
>>> So there are a number of tradeoffs. If for example, Sakai 2.6.0 has hypothetically 100 bugs, and 50 bugfixes are merged from trunk to 2-6-x, and there's a regression risk of 10% for merges which means that 5 new bugs are introduced, is 2-6-x better off than 2.6.0 ? Numerically yes, as it would have 50 + 5 = 55 bugs rather than 100. However, those 10% regressions are potentially 5 new bugs in previously bug-free areas of code. 
>>> 
>>> There is a case that having known bugs is better than the risk of continuous exposure to new unknown bugs, and I believe this is largely behind the decision of sites that run tags rather than maintenance branches.
>>> 
>>> My view on the balance of responsibilities between developers and branch managers is that if we make the bar too high for branch managers, (a) the time cost of merging an issue goes way up, (b) we will have even fewer branch managers. So IMO developers need to have more end-to-end responsibilities for their issues and the likely impact on maintenance branches.
>>> 
>>> I'm also going to venture an opinion that we should reduce the number of issues that are flagged for merging to maintenance branches, as clearly we are not able to keep up with the existing volume in a meaningful way. For example, we could adopt a new practice that only bugfixes rated Critical or Blocker should get flagged for maintenance branches. 
>>> 
>>> We should also release maintenance tags more frequently (e.g. every 3 months), and keep the volume of changes low enough that meaningful QA can be done on the tags before release.
>>> 
>>> Regards
>>> Stephen
>>> 
>>>>>> "Berg, Alan" <A.M.Berg at uva.nl> 8/5/2010 9:01 AM >>> 
>>> Hi all,
>>> 
>>> This is an interesting debate that should be discussed further. Both side's make valid points.
>>> 
>>> I missed most of the debate, but I  support Steve S's view that " all fixes that go into a maintenance branch must be tested by the person doing that work, before they are committed."  QA does not have enough resources to have a different perspective.
>>> 
>>> At the moment we have a mild explosion of tick boxes in Jira. Perhaps we should add one/two for QA testing requested, code review requested, but that would simply slow down the lifecycle. It is a difficult balance.
>>> 
>>> Alan B
>>> 
>>> Alan Berg
>>> QA Director - The Sakai Foundation
>>> 
>>> Senior Developer / Quality Assurance
>>> Group Education and Research Services
>>> Central Computer Services
>>> University of Amsterdam
>>> 
>>> http://home.uva.nl/a.m.berg
>>> ________________________________________
>>> From: sakai-qa-bounces at collab.sakaiproject.org [sakai-qa-bounces at collab.sakaiproject.org] on behalf of Steve Swinsburg [steve.swinsburg at gmail.com]
>>> Sent: 05 August 2010 08:46
>>> To: Stephen Marquard
>>> Cc: David Horwitz; sakai-qa at collab.sakaiproject.org; Alan Berg
>>> Subject: Re: [WG: Sakai QA] 2.7 webservice broken
>>> 
>>> I'm not debating that the issue wasn't tested at all, as the comments in the JIRA show that it was (or at least verified).
>>> 
>>> The issue is that it was committed to the branch without further testing to ensure that the fix was appropriate for that branch. That's the crux of my argument, that all fixes that go into a maintenance branch must be tested by the person doing that work, before they are committed.
>>> 
>>> cheers,
>>> Steve
>>> 
>>> 
>>> On 05/08/2010, at 4:08 PM, Stephen Marquard wrote:
>>> 
>>>> Maybe this issue is a victim of no two people having the same understanding of JIRA processes.
>>>> 
>>>> On this particular issue, the comments indeed have evidence of testing, in the form of "Confirmed fixed in trunk." So whether or not the testing was accurate or not, the branch manager followed the process correctly.
>>>> 
>>>> As it seems like an interesting debate, CC'ing the QA list.
>>>> 
>>>> Regards
>>>> Stephen
>>>> 
>>>>>>> Steve Swinsburg <steve.swinsburg at gmail.com> 8/5/2010 7:57 AM >>>
>>>> I'm alarmed by this and quite strongly disagree.
>>>> 
>>>> It's been said time and time again that one of the duties of a branch manager is to test every single thing that goes into that branch. It even says so here: http://confluence.sakaiproject.org/display/REL/Branch+Manager+Guidelines under 'When you have a fix you want to merge'.
>>>> 
>>>> Yes, there is a part that says "Verified should be ok to merge but check the comments for evidence... (of) testing", so read on.
>>>> 
>>>> So the issue had the merge status set and the branch manager picked it up. That doesn't mean it should be blindly committed without being tested first. Anyone could easily go into JIRA and update a closed issue to add a merge status for the branches. Under this process, a branch manager could come along and merge a fix that breaks things, simply because it wasn't tested. And this might not even be picked up until several tags later.
>>>> 
>>>> The branches need to be as stable as a rock, and that means testing everything before it's committed.
>>>> 
>>>> I'd be surprised if this position wasn't echoed by the QA team.
>>>> 
>>>> thanks,
>>>> Steve
>>>> 
>>>> 
>>>> 
>>>> On 05/08/2010, at 3:42 PM, Stephen Marquard wrote:
>>>> 
>>>>> From the issue change log, it appears it was created with a 2-7-x status field set to Merge. As it was then Resolved and later Closed by Sam Ottenhof, the branch manager (Jonathan Cook) in this case acted correctly in terms of process by merging it to 2-7-x.
>>>>> 
>>>>> We cannot reasonably expect branch managers to do functional testing on changes that they merge - that's why the process is to merge Closed issues rather than Resolved issues, and to rely on developers to set the 2-6-x and 2-7-x status correctly in the first place.
>>>>> 
>>>>> Regards
>>>>> Stephen
>>>>> 
>>>>>>>> Steve Swinsburg <steve.swinsburg at gmail.com> 8/5/2010 12:35 AM >>>
>>>>> Hi Dave,
>>>>> 
>>>>> This http://jira.sakaiproject.org/browse/SAK-18776 should never have been merged to 2.7.
>>>>> 
>>>>> It's also obvious that this wasn't tested before being committed which is worrying. If it was, it would be clear that it didn't compile.  Perhaps we need to send out a reminder that all changes in a branch must be thoroughly tested before being committed?
>>>>> 
>>>>> I'll fix this.
>>>>> 
>>>>> cheers,
>>>>> S
>>>>> 
>>>>> 
>>>>> 
>>>>> On 04/08/2010, at 11:26 PM, David Horwitz wrote:
>>>>> 
>>>>>> Hi Steve,
>>>>>> 
>>>>>> It seems compilation of SakaiScript is broken in 2.7. It seems that a number of issues related to:
>>>>>> 
>>>>>> http://jira.sakaiproject.org/browse/KNL-428
>>>>>> 
>>>>>> Have been merged to 2.7 but KNL-428 is not in the latest 1.1 kernel (it was never flagged for inclusion). It seems we either have to:
>>>>>> 
>>>>>> 1) Revert the WS merges
>>>>>> 2) Merge KNL-428 and release a new kernel
>>>>>> 
>>>>>> 
>>>>>> Advice?
>>>>>> 
>>>>>> D
>>>>> 
>>>>> 
>>>>> 
>>>>> 
>>>>> 
>>>>> 
>>>>> ###
>>>>> UNIVERSITY OF CAPE TOWN
>>>>> 
>>>>> This e-mail is subject to the UCT ICT policies and e-mail disclaimer published on our website at http://www.uct.ac.za/about/policies/emaildisclaimer/ or obtainable from +27 21 650 4500. This e-mail is intended only for the person(s) to whom it is addressed. If the e-mail has reached you in error, please notify the author. If you are not the intended recipient of the e-mail you may not use, disclose, copy, redirect or print the content. If this e-mail is not related to the business of UCT it is sent by the sender in the sender's individual capacity.
>>>>> 
>>>>> ###
>>>>> 
>>>> 
>>>> 
>>>> 
>>>> 
>>>> 
>>>> 
>>>> ###
>>>> UNIVERSITY OF CAPE TOWN
>>>> 
>>>> This e-mail is subject to the UCT ICT policies and e-mail disclaimer published on our website at http://www.uct.ac.za/about/policies/emaildisclaimer/ or obtainable from +27 21 650 4500. This e-mail is intended only for the person(s) to whom it is addressed. If the e-mail has reached you in error, please notify the author. If you are not the intended recipient of the e-mail you may not use, disclose, copy, redirect or print the content. If this e-mail is not related to the business of UCT it is sent by the sender in the sender's individual capacity.
>>>> 
>>>> ###
>>>> 
>>> 
>>> _______________________________________________
>>> sakai-qa mailing list
>>> sakai-qa at collab.sakaiproject.org
>>> http://collab.sakaiproject.org/mailman/listinfo/sakai-qa
>>> 
>>> TO UNSUBSCRIBE: send email to sakai-qa-unsubscribe at collab.sakaiproject.org with a subject of "unsubscribe"
>>> 
>>> 
>>> 
>>> 
>>> 
>>> ###
>>> UNIVERSITY OF CAPE TOWN 
>>> 
>>> This e-mail is subject to the UCT ICT policies and e-mail disclaimer published on our website at http://www.uct.ac.za/about/policies/emaildisclaimer/ or obtainable from +27 21 650 4500. This e-mail is intended only for the person(s) to whom it is addressed. If the e-mail has reached you in error, please notify the author. If you are not the intended recipient of the e-mail you may not use, disclose, copy, redirect or print the content. If this e-mail is not related to the business of UCT it is sent by the sender in the sender's individual capacity.
>>> 
>>> ###
>>> 
>>> _______________________________________________
>>> sakai-qa mailing list
>>> sakai-qa at collab.sakaiproject.org
>>> http://collab.sakaiproject.org/mailman/listinfo/sakai-qa
>>> 
>>> TO UNSUBSCRIBE: send email to sakai-qa-unsubscribe at collab.sakaiproject.org with a subject of "unsubscribe"
>> 
>> _______________________________________________
>> sakai-qa mailing list
>> sakai-qa at collab.sakaiproject.org
>> http://collab.sakaiproject.org/mailman/listinfo/sakai-qa
>> 
>> TO UNSUBSCRIBE: send email to sakai-qa-unsubscribe at collab.sakaiproject.org with a subject of "unsubscribe"
> 
> _______________________________________________
> sakai-qa mailing list
> sakai-qa at collab.sakaiproject.org
> http://collab.sakaiproject.org/mailman/listinfo/sakai-qa
> 
> TO UNSUBSCRIBE: send email to sakai-qa-unsubscribe at collab.sakaiproject.org with a subject of "unsubscribe"
> 
> 

-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://collab.sakaiproject.org/pipermail/sakai-qa/attachments/20100805/6b5e431c/attachment-0001.html 
-------------- next part --------------
A non-text attachment was scrubbed...
Name: smime.p7s
Type: application/pkcs7-signature
Size: 3829 bytes
Desc: not available
Url : http://collab.sakaiproject.org/pipermail/sakai-qa/attachments/20100805/6b5e431c/attachment-0001.bin 


More information about the sakai-qa mailing list