[sakai2-tcc] Thoughts on 2.9.1 schedule

[Matthew Jones]

Nothing against Anthony, but I'm not sure why everyone is so eager to talk
to him about the release process. He hasn't been involved in well over a
year, the last 2.8.2 release being done by Steve and the 2.9 releases being
done by Neal, Sam, Chris and myself.

I'd say the process is entirely changed from release to QA for 2.8, all
adding additional time and complexity to the process. Some of this might be
for the better, some for the worse.

   - In 2.8, we didn't do any QA verification on any tickets before
   merging. If the ticket was fixed it was merged. That workflow step did not
   exist. There were functional tests fests and bug bashes, but much of the
   delay in 2.9 release was the in-depth testing on individual issues as well
   as the much larger amount of testers in general. As mentioned many times
   before, the number of issues created lately are far out pacing the number
   of issues fixed.
   - Solution - We need more developers contributing on software
      maintenance. Though I don't know how to actually resolve this.
Back in 2.8
      the "maintenance team" had almost 15-20 active members. Now it's
about 5-10.
   - In 2.8 Jenkins was only building 6 indiesand all releases were done on
   the command line. In 2.9 it's building all 29 indies and being used to
   release them all.
   - Solution(s) - A lot of possibilities here.

1) Reduce the number of indies, moving more of them back into core. Core is
easier to release than the indies and most of these will never be released
as indies.
2) Split the api's from the impl/tool and release the apis at the beginning
of the cycle as a 2 digit version (2.9). This will make the tool/impl
easier to release because order wouldn't matter as the required
dependencies would already be available. We'd also change the versions in
master/kernel a LOT less often. We'd probably also get rid of the
assemblies and pack because I doubt many people would miss them.
  - The assemblies aren't useful for the binary/demo since everything will
be there. The pack won't be useful for the source since I believe almost
everyone now just builds off and prefers the -all branch. We're not
releasing most contrib tools that people could actually use, but with the
static api dependencies in maven central these are trivial to build and we
could even have an "on-demand" server that could create zips.
  - Obviously these would include a few more things (like kernel-util) but
these were refactored so they shouldn't change as much (at all) since the
impl isn't in this anymore.
3) Get rid of Jenkins, just do the release on the command line. Might make
the release process a lot faster overall. Might be worth combining with 1 &
2. Perhaps leaving Jenkins building SNAPSHOTs for verification, but I don't
even know if publishing snapshot artifacts is good as that can mess up
local builds, especially for people using Maven 2 that aren't compatible
with the newer snapshot style

   - In 2.8, Anthony had dedicated time to work on the release
   (merging/releasing) that wasn't subject to local interruption. Currently
   Neal has that but isn't skilled enough to do it. Anthony wasn't doing it
   starting out either and kind of took it on after some time working, and it
   really wasn't part of the initial job description for Neal either.
   - Solution - We'll either need to get Neal up to speed or find some way
      to fund dedicated time for others to do it so it can be easier scheduled
      against local interests. Training Neal has a cost of time and
money, and is
      a long term process. This is something we'll probably need to
discuss more
      F2F at the conference.

On Sat, Jan 19, 2013 at 3:38 AM, Berg, Alan <A.M.Berg at uva.nl> wrote:

>  I have an online meeting with Anthony next week. I will see if I can
> help improve the deterministicness of the release automation. Anyone else
> is welcome.
>
>  Sounds like the use of the word padding was unfortunate. Hidden
> bottlenecks and all. Keep up up the good/hard work.
>
>
>  Regards Alan,
>
> Sent from somewhere interesting via a Mobile device.
>
>
>
> -------- Original message --------
> Subject: Re: [sakai2-tcc] Thoughts on 2.9.1 schedule
> From: Matthew Jones <matthew at longsight.com>
> To: Neal Caidin <nealcaidin at sakaifoundation.org>
> CC: "sakai2-tcc at collab.sakaiproject.org 2" <
> sakai2-tcc at collab.sakaiproject.org>
>
>
>  TL;DR - Release process is too complex, local issues are part of the
> delay, perhaps we should just skip release candidates for minor releases
> like we did in the past and put out 2.9.1.
>
>  I haven't caught up in depth on this is entire thread as this is my
> first day back from Phoenix and still jet lagged. However nobody who
> commented besides Neal regularly puts any predictable effort into the
> actual release process. And as far as the process of actually cutting the
> release and merging the tickets, myself and Sam currently do the vast
> majority of that work. So the schedule is very dependent on our local
> workload. This week I was out almost all week catching up doay, and Sam
> will be out most of next week, putting some extra local pressure on me.
> Some of these delays are built into the overly padded schedule that we've
> kept Neal updated on.
>
>  However some of the padding is just because of the actual known
> difficulty and anticipated delays from the past 3 months in the time
> required to perform the actual release and getting people on-board with the
> QA effort when needed so we're ready for the next cycle. We could smoke
> test on nightly and just skip rc01 going straight to 2.9.1 and for sure
> save a lot of time cutting the schedule in half. This was what we did in
> the past for previous minor releases (2.7.1, 2.8.1, 2.8.2), just verifying
> if the software actually starts up.
>
>  Some known delays and thoughts.
>
>  - CLE Release Team currently all meet once every Thursday morning, which
> generally puts at *least* a week between making any actual decisions. We
> could make decisions via email asynchronously with time limits, but this
> doesn't necessarily mean things will happen faster. We could potentially
> meet more often when the release is about to happen, but it's difficult to
> time times where all interested parties can meet.
>
>  - Currently, as mentioned, Sam and I are important to the process of
> both branching and releasing. We're happy to distributed this more but the
> only person who's done any major branching lately is Greg at IU, though not
> consistently, and he's seems busier now. 2.9 has 63 issues waiting to
> merge, 2.8 has 65, most of these verified. Most of these will be pushed to
> 2.9.2 or 2.9.3 even though they're probably good fixes.  Steve Swinsburg
> does help on the releases (especially 2.8), but he hasn't done a full
> release of 2.9.
>
>  - The release process for 2.9 in general is overly complex. When Anthony
> was releasing we had half of the tools set up as indies as we do now and
> everything was done on command line. Now it's all done through Jenkins. The
> idea with this is it would make it easier for other people to do off cycle
> releases (Like Bryan does occasionally since I've showed him how to do it
> last year at the conference, and Steve does occasionally) but really just
> introduces random complexity because the Jenkins M2 Release Plugin has a
> lot of bugs, and Maven still has a lot of missing capability as well.
>   There's going to need to be a separate email addressing this, but I'm
> almost reluctant to do *any* releases both because of the actual time it
> takes (2-3 hours one day, waiting ~8 hours for Jenkins, then 2-3 hours the
> next day) as well as the confusion it ends up causing those who use msub to
> customize builds and anyone who uses the 2.9.x branch. When versions get
> bumped, the SNAPSHOT version also gets dumped which REQUIRES that change be
> pushed through the entire local code-base. When Jenkins updates it often
> does something new and unpredictable each release cycle that I get to
> experience and attempt to figure out! Great fun! :)
>
>  Much of this is stuff we'd like to have Neal do, and it's part of his 3+
> month on-boarding, but it really requires him to go some where and get
> training from someone that knows it (or someone coming to him). Doing this
> over Skype isn't very convenient.  I have plans for this to happen (If I'm
> going to be training again) by at least April. Though I'm not sure with the
> way the release process works and the randomness of Jenkins that it would
> be possible for someone who doesn't have a lot of experience
> troubleshooting maven errors (Something I've wrangling with years) would be
> able to solve easily.
>
>  On Fri, Jan 18, 2013 at 8:27 AM, Neal Caidin <
> nealcaidin at sakaifoundation.org> wrote:
>
>> Yes, no new blockers regressions, and have a fairly high bar for a show
>> stopper. I haven't checked this morning to see if they have been
>> fixed/merged, but we do have a few security issues that possibly won't get
>> done in time.
>>
>> I'm at EuroSakai the week of Jan 28, that might make the coordination
>> tricky. Unlike Educause where I was barely a blip, I expect I will be
>> getting a lot of attention in Paris and will be hard pressed to find time
>> for QA coordination/check-in. Would be nice to have a backup QA lead, but I
>> don't.
>>
>> So maybe the schedule range is more of the first or second week in
>> February. But if we need an RC02, then that schedule fails, correct? And I
>> need to have a QA lead/check in backup plan. If I don't have that, it could
>> delay things a week and we are back to a mid-February.
>>
>> - Neal
>>
>>
>>
>>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://collab.sakaiproject.org/pipermail/sakai2-tcc/attachments/20130119/0bca8bc0/attachment-0001.html