[WG: Sakai QA] [Building Sakai] Jira process clarification and modification - "Target Version"

Fri Aug 21 13:59:47 PDT 2009

I agree with Anthony and it follows with the idea of having QA happen earlier in the process. One reservation I have concerns what I've heard repeatedly concerning code that was rushed in to the release at the last minute and then found to have regressions. I'm not sure how this would be addressed in a model like this. Additionally, are the developers then responsible for all affected branch merges (2.5.x, 2.6.x, 2.7.x)?

Option
* Perhaps, as in your proposal, we have a temporary moratorium on 2.6.x gatekeeping to clear fixed and QAed issues waiting in the branch lobby.  
* Then we close the gate again to cut test tags, QA the integrated branch, and report and address any regressions.
* Open the gates
* Repeat

NOTE: These scenarios have the potential of increasing the errors and regressions in a release. Not all commiters will have the same skills and abilities and it is almost certain we will encounter problems. On the otherhand, we will also be merging in many more "fixed" issues, so the larger question is would we have encounter similar problems if these issues were merged by a single BM with really fast fingers.  

Just some thoughts,

Pete

> -----Original Message-----
> From: sakai-dev-bounces at collab.sakaiproject.org [mailto:sakai-dev-
> bounces at collab.sakaiproject.org] On Behalf Of Anthony Whyte
> Sent: Friday, August 21, 2009 1:00 PM
> To: Michael Korcuska
> Cc: QA Sakai; Sakai Developers
> Subject: Re: [Building Sakai] [WG: Sakai QA] Jira process clarification
> and modification - "Target Version"
> 
> It might do well to unpack the discussion still further and ask if our
> current assumptions regarding the value of the gate keeping embedded in
> the workflow outlined below still makes sense.  Branch management gate
> keeping, for instance, is currently practiced as a part-time occupation
> by a few hardy community souls who volunteer their time to merge fixes
> into our *.x maintenance branches.  This is an important task without
> which we could roll no releases.
> 
> Unfortunately, verified fixes ready for merging are often forced to
> queue up for rather long periods of time while the few branch managers
> we have available labor to process merge requests while juggling their
> other commitments.  Many of these merge requests could in fact be
> handled by the originating project teams themselves with little risk to
> the target maintenance branch.  Tested and verified fixes could be
> brought to market faster and branch managers could instead focus on
> general branch oversight and coordination activities.  If, as in the
> case of 2.6.x, we raised the portcullis currently defending the
> maintenance branch and permitted project teams to merge their own
> tested and verified fixes we could reduce dramatically the current
> backlog of fixes queued up to be merged into 2.6.x (170+ issues by my
> count).
> 
> Branch oversight is important but if we practice it in a way that
> impedes the otherwise quick delivery of verified and tested fixes we
> gain no advantage.
> 
> Cheers,
> 
> Anth
> 
> 
> On Aug 21, 2009, at 11:18 AM, Michael Korcuska wrote:
> 
> > I think this is a good discussion. I think we need to take a bit more
> > time and that some confusion is resulting from the fact that we don't
> > have a common understanding of what the work flow is (or should be).
> > So I recommend starting with a plain English (no reference to Jira)
> > description of what should be happening. Once we agree on that then
> we
> > can figure out how to wrestle jira into helping us reflect that as
> > simply as possible.  It would also serve as good documentation for
> > newcomers.
> >
> > For bug something like:
> >
> > Someone identifies issue and the version(s) of the software it
> affects
> > Others verify the issue exists and examine whether it affects other
> > releases/trunk Some developer takes responsibility for the issue and
> > lets everyone know it is being worked on Developer describes fix on
> > list and asks for approval to commit (possibly optional) Developer
> > submits a fix to the relevant branch and asks for testing QA tests
> the
> > issue and either sends it back for more work or confirms it is fixed
> > Branch managers apply fix to their branches, resolving merge
> conflicts
> > along the way Fix is verified in the relevant branches
> >
> > I know this probably isn't exactly right, but I hope you get the
> idea.
> > There are other complexities like "I was working on this but now I'm
> > not" that might be relevant.
> >
> > The crux of the matter, I think, happens after a fix is created for a
> > particular branch, say trunk. In some sense, from the developers
> > perspective, that closes the issue. But there may still be work to
> > apply that issue to multiple branches. And so a single bug turns
> > into multiple issues to track from a QA and release perspective.
> > The branch managers and release team want to be able to figure out
> not
> > only if a bug was fixed but whether that fix was applied to their
> > branch.
> >
> > Thoughts?
> >
> > Michael
> >
> > On Aug 21, 2009, at 07:17, Anthony Whyte wrote:
> >
> >> I should note that if I was going to shrink the list of statuses
> >> below (I like as few steps as possible) it would go with:
> >>
> >> Open
> >> Test
> >> Merge
> >> Closed
> >> Reopened
> >> Suspended
> >>
> >> Cheers,
> >>
> >> Anth
> >>
> >>
> >> On Aug 21, 2009, at 10:04 AM, Anthony Whyte wrote:
> >>
> >>> I'd like to see the target version restored for planning,
> scheduling
> >>> and as an indicator of where finished/fixed code needs to be merged
> >>> (rather than the current custom *.x status dropdowns).  This would
> >>> be coupled with a revised status workflow that would include steps
> >>> resembling the following:
> >>>
> >>> ___________________
> >>>
> >>> Open
> >>>
> >>> Work in progress ("in progress" does exist) Work suspended Work
> >>> resumed [perhaps redundant, just reset to "work in progress"
> >>> if previously suspended]
> >>>
> >>> [Review requested
> >>> Review started
> >>> Review completed]  [perhaps a set of "review" phase statuses is
> >>> overkill (could be construed as part of "work" above) but then
> again
> >>> . . .]
> >>>
> >>> Testing requested
> >>> Testing started
> >>> Testing completed
> >>>
> >>> Merge requested (this step might involve merging to multiple
> >>> branches)
> >>> Merge started  [not sure we'd ever use this one] Merge completed
> >>>
> >>> Closed [means done, fini, for the history books, all workflow
> >>> actions completed, time for a Theakston's Old Peculiar or sleep]
> >>>
> >>> Reopened
> >>>
> >>> ________________________
> >>>
> >>> The K2 team has developed a set of workflow statuses similar to
> this
> >>> and I like how they are approaching the challenge of tracking tasks
> >>> through their life cycle; although I should note that they do not
> >>> use a target version.
> >>>
> >>> So restore the target version, revise the 2.x workflow, eliminate
> >>> the custom *.x branch status fields -- made redundant by
> restoration
> >>> of target version and revised workflow statuses and continue work
> on
> >>> streamlining and simplifying as much as is possible our issue
> >>> tracking workflows.
> >>>
> >>> By the way, it's not hard to check the change history to check the
> >>> workflow on an issue and ensure that, for example testing has been
> >>> completed before a branch manager or myself honor a merge request
> >>> but I'd need the target version set correctly to know where I
> should
> >>> merge the code (acknowledging of course that it's usually
> >>> obvious) without those pesky custom *.x status dropdowns (which I
> >>> would like to see eliminated).  Alongside this the assignee could/
> >>> should be updated to reflect the person/team responsible for a
> given
> >>> phase of work.  Megan I believe has configured IU's Jira to list
> >>> multiple assignees--something else to look at.
> >>>
> >>> Finally, we should create a Jira cheat sheet so that people who use
> >>> Sakai's issue tracking system can quickly get themselves up to
> speed
> >>> on how it all works.
> >>>
> >>> Cheers,
> >>>
> >>> Anth
> >>>
> >>>
> >>> On Aug 21, 2009, at 8:41 AM, Noah Botimer wrote:
> >>>
> >>>> Matthew,
> >>>>
> >>>> Thanks for the response. Notes inline below.
> >>>>
> >>>> On Aug 21, 2009, at 6:36 AM, Matthew Buckett wrote:
> >>>>
> >>>>> 2009/8/21 Noah Botimer <botimer at umich.edu>:
> >>>>>>
> >>>>>> This goes to a concrete suggestion, rather than to an outright
> >>>>>> redefinition of your goal. Three principles first:
> >>>>>
> >>>>> Completely agree with your principals.
> >>>>>
> >>>>>> To meet with these principles, I suggest that we evaluate the
> >>>>>> following for
> >>>>>> fitness:
> >>>>>> 1. Favoring subtasks (which may be assigned) over our merge
> >>>>>> status fields when something is fixed in trunk and should be
> >>>>>> included in a maintenance release
> >>>>>
> >>>>> I worry that this may make is harder to find if an issue has been
> >>>>> fixed in a branch as you are more reliant on using JIRA to locate
> >>>>> the SAK for the branch rather than an issue having the same SAK
> >>>>> across all branches and msub spaces.
> >>>>
> >>>> I am somewhat concerned about this myself but I'm not sure if it's
> >>>> as big as we might imagine. At worst, we could use guidelines like
> >>>> including both numbers in commit comments. At best, it might
> >>>> clarify the logs some by avoiding the large numbers of "duplicate"
> >>>> changes tracked on a ticket (if we didn't include both numbers). i
> >>>> haven't formed a final opinion here.
> >>>>
> >>>>>
> >>>>>> 3. Guiding our contributor community in setting Fix Version
> >>>>>> appropriately as a target for unresolved issues and allowing
> this
> >>>>>> to default into zero-touch accuracy when following through to
> >>>>>> actual resolution
> >>>>>
> >>>>> Sorry I don't follow, could you explain what you mean by this
> >>>>
> >>>> I mean that we need a clear definition of what the "right" Fix
> >>>> Version is for intended outcomes. If we follow through on our
> >>>> intentions, there is no post-management of the ticket required.
> >>>> Tickets and filters would become more stable over the release
> cycle
> >>>> and be automatically correct in the nominal case (file w/targeted
> >>>> version(s), commit, verify, close ticket).
> >>>>
> >>>> The QA/release task becomes verifying the work and only changing
> >>>> versions on the tickets that were wrong (missed a release,
> >>>> indicated by an unresolved status; wrong version; had to roll
> >>>> back). We shouldn't elect for setting things we know someone will
> >>>> have to fix down the line, but prefer cases where someone applies
> >>>> minimal or no change when the right things were done before them
> >>>> (e.g., closing the ticket with a verification comment).
> >>>>
> >>>> In our community, the folks closest to the work are pretty well
> >>>> attuned to what should be targeted where. I have faith that the
> >>>> core workers on an issue can get it right in the vast majority,
> >>>> provided there is a clear process and a very simple cheat sheet.
> >>>>
> >>>> Thanks,
> >>>> -Noah
> >>>>
> >>>> _______________________________________________
> >>>> sakai-dev mailing list
> >>>> sakai-dev at collab.sakaiproject.org
> >>>> http://collab.sakaiproject.org/mailman/listinfo/sakai-dev
> >>>>
> >>>> TO UNSUBSCRIBE: send email to
> >>>> sakai-dev-unsubscribe at collab.sakaiproject.org
> >>>>  with a subject of "unsubscribe"
> >>>>
> >>>>
> >>>
> >>> _______________________________________________
> >>> sakai-dev mailing list
> >>> sakai-dev at collab.sakaiproject.org
> >>> http://collab.sakaiproject.org/mailman/listinfo/sakai-dev
> >>>
> >>> TO UNSUBSCRIBE: send email to
> >>> sakai-dev-unsubscribe at collab.sakaiproject.org
> >>>  with a subject of "unsubscribe"
> >>
> >> _______________________________________________
> >> sakai-qa mailing list
> >> sakai-qa at collab.sakaiproject.org
> >> http://collab.sakaiproject.org/mailman/listinfo/sakai-qa
> >>
> >> TO UNSUBSCRIBE: send email to
> >> sakai-qa-unsubscribe at collab.sakaiproject.org
> >>  with a subject of "unsubscribe"
> >
> > --
> > Michael Korcuska
> > Executive Director, Sakai Foundation
> > mkorcuska at sakaifoundation.org
> > phone: +1 510-859-4247 (google voice)
> > skype: mkorcuska
> >
> >
> >
> >
> >
> >