[sakai2-tcc] Thoughts on 2.9.1 schedule

[Anthony Whyte]

Only Alan Berg has pinged me about chatting about release stuff.  I always enjoy talking to Alan but I was surprised to see notice of our upcoming conversation inserted into this thread.  It suggested--wrongly in my view--that our talk would have some bearing on the current release cycle.  I can't speak for Alan's availability but my new responsibilities leave me with little or no time to participate actively in either CLE branch management or release activities.  The best I can do at present is work the crowd, seeing if I can garner commitments to assist in one way or another, as occurred with the Michigan offer to participate in a RC1 test fest if one is organized.  I don't raise concerns without trying in some way to help.

But back to 2.9.1.  Expectations were raised that schools would see a follow up maintenance release soon after the release of 2.9.0 last November.  The release has yet to be cut.  After participating in several TCC and release team calls both before and after the holidays I got the sense that 2.9.1 might not see the light of day before mid-February or early March.  I happen to think that 2.9.1 is fairly ripe for release and that the release schedule should be tightened down a bit in order to get it out the door in the next three weeks.  If those doing the work disagree in this case, then so be it.    

Regarding Matt's suggestions (some of which are by no means new) I don't believe many are going to help 2.9.1, indeed perhaps not even 2.9.x--except the idea of going back to the command line to do the release.  That's not to say that Matt's ideas are wrong, I just think they obscure the key challenge for 2.9.1 and future releases--ensuring that the scope of the release is aligned to the available resources at hand, however many, however few.

One final observation: it looks to me like Longsight has been left to shoulder the twin responsibilities of 2.9.x branch management and release work alone.  If true, it's both unfair and wrong.

Cheers,

Anth


P.S.  for the record, the 2.8.0 release included 22 indie projects (not 6).  As for the now defunct maintenance team, it never had 15-20 truly active members.


On Jan 19, 2013, at 12:52 PM, Matthew Jones wrote:

> Nothing against Anthony, but I'm not sure why everyone is so eager to talk to him about the release process. He hasn't been involved in well over a year, the last 2.8.2 release being done by Steve and the 2.9 releases being done by Neal, Sam, Chris and myself. 
> 
> I'd say the process is entirely changed from release to QA for 2.8, all adding additional time and complexity to the process. Some of this might be for the better, some for the worse.
> In 2.8, we didn't do any QA verification on any tickets before merging. If the ticket was fixed it was merged. That workflow step did not exist. There were functional tests fests and bug bashes, but much of the delay in 2.9 release was the in-depth testing on individual issues as well as the much larger amount of testers in general. As mentioned many times before, the number of issues created lately are far out pacing the number of issues fixed.
> Solution - We need more developers contributing on software maintenance. Though I don't know how to actually resolve this. Back in 2.8 the "maintenance team" had almost 15-20 active members. Now it's about 5-10.
> In 2.8 Jenkins was only building 6 indiesand all releases were done on the command line. In 2.9 it's building all 29 indies and being used to release them all.
> Solution(s) - A lot of possibilities here. 
> 1) Reduce the number of indies, moving more of them back into core. Core is easier to release than the indies and most of these will never be released as indies.
> 2) Split the api's from the impl/tool and release the apis at the beginning of the cycle as a 2 digit version (2.9). This will make the tool/impl easier to release because order wouldn't matter as the required dependencies would already be available. We'd also change the versions in master/kernel a LOT less often. We'd probably also get rid of the assemblies and pack because I doubt many people would miss them.
>   - The assemblies aren't useful for the binary/demo since everything will be there. The pack won't be useful for the source since I believe almost everyone now just builds off and prefers the -all branch. We're not releasing most contrib tools that people could actually use, but with the static api dependencies in maven central these are trivial to build and we could even have an "on-demand" server that could create zips.
>   - Obviously these would include a few more things (like kernel-util) but these were refactored so they shouldn't change as much (at all) since the impl isn't in this anymore.
> 3) Get rid of Jenkins, just do the release on the command line. Might make the release process a lot faster overall. Might be worth combining with 1 & 2. Perhaps leaving Jenkins building SNAPSHOTs for verification, but I don't even know if publishing snapshot artifacts is good as that can mess up local builds, especially for people using Maven 2 that aren't compatible with the newer snapshot style
> In 2.8, Anthony had dedicated time to work on the release (merging/releasing) that wasn't subject to local interruption. Currently Neal has that but isn't skilled enough to do it. Anthony wasn't doing it starting out either and kind of took it on after some time working, and it really wasn't part of the initial job description for Neal either.
> Solution - We'll either need to get Neal up to speed or find some way to fund dedicated time for others to do it so it can be easier scheduled against local interests. Training Neal has a cost of time and money, and is a long term process. This is something we'll probably need to discuss more F2F at the conference.
> On Sat, Jan 19, 2013 at 3:38 AM, Berg, Alan <A.M.Berg at uva.nl> wrote:
> I have an online meeting with Anthony next week. I will see if I can help improve the deterministicness of the release automation. Anyone else is welcome.
> 
> Sounds like the use of the word padding was unfortunate. Hidden bottlenecks and all. Keep up up the good/hard work.
> 
> 
> Regards Alan,
> 
> Sent from somewhere interesting via a Mobile device.
> 
> 
> 
> -------- Original message --------
> Subject: Re: [sakai2-tcc] Thoughts on 2.9.1 schedule 
> From: Matthew Jones <matthew at longsight.com> 
> To: Neal Caidin <nealcaidin at sakaifoundation.org> 
> CC: "sakai2-tcc at collab.sakaiproject.org 2" <sakai2-tcc at collab.sakaiproject.org> 
> 
> 
> TL;DR - Release process is too complex, local issues are part of the delay, perhaps we should just skip release candidates for minor releases like we did in the past and put out 2.9.1. 
> 
> I haven't caught up in depth on this is entire thread as this is my first day back from Phoenix and still jet lagged. However nobody who commented besides Neal regularly puts any predictable effort into the actual release process. And as far as the process of actually cutting the release and merging the tickets, myself and Sam currently do the vast majority of that work. So the schedule is very dependent on our local workload. This week I was out almost all week catching up doay, and Sam will be out most of next week, putting some extra local pressure on me. Some of these delays are built into the overly padded schedule that we've kept Neal updated on.
> 
> However some of the padding is just because of the actual known difficulty and anticipated delays from the past 3 months in the time required to perform the actual release and getting people on-board with the QA effort when needed so we're ready for the next cycle. We could smoke test on nightly and just skip rc01 going straight to 2.9.1 and for sure save a lot of time cutting the schedule in half. This was what we did in the past for previous minor releases (2.7.1, 2.8.1, 2.8.2), just verifying if the software actually starts up. 
> 
> Some known delays and thoughts.
> 
> - CLE Release Team currently all meet once every Thursday morning, which generally puts at *least* a week between making any actual decisions. We could make decisions via email asynchronously with time limits, but this doesn't necessarily mean things will happen faster. We could potentially meet more often when the release is about to happen, but it's difficult to time times where all interested parties can meet.
> 
> - Currently, as mentioned, Sam and I are important to the process of both branching and releasing. We're happy to distributed this more but the only person who's done any major branching lately is Greg at IU, though not consistently, and he's seems busier now. 2.9 has 63 issues waiting to merge, 2.8 has 65, most of these verified. Most of these will be pushed to 2.9.2 or 2.9.3 even though they're probably good fixes.  Steve Swinsburg does help on the releases (especially 2.8), but he hasn't done a full release of 2.9. 
> 
> - The release process for 2.9 in general is overly complex. When Anthony was releasing we had half of the tools set up as indies as we do now and everything was done on command line. Now it's all done through Jenkins. The idea with this is it would make it easier for other people to do off cycle releases (Like Bryan does occasionally since I've showed him how to do it last year at the conference, and Steve does occasionally) but really just introduces random complexity because the Jenkins M2 Release Plugin has a lot of bugs, and Maven still has a lot of missing capability as well. 
>   There's going to need to be a separate email addressing this, but I'm almost reluctant to do *any* releases both because of the actual time it takes (2-3 hours one day, waiting ~8 hours for Jenkins, then 2-3 hours the next day) as well as the confusion it ends up causing those who use msub to customize builds and anyone who uses the 2.9.x branch. When versions get bumped, the SNAPSHOT version also gets dumped which REQUIRES that change be pushed through the entire local code-base. When Jenkins updates it often does something new and unpredictable each release cycle that I get to experience and attempt to figure out! Great fun! :)
> 
> Much of this is stuff we'd like to have Neal do, and it's part of his 3+ month on-boarding, but it really requires him to go some where and get training from someone that knows it (or someone coming to him). Doing this over Skype isn't very convenient.  I have plans for this to happen (If I'm going to be training again) by at least April. Though I'm not sure with the way the release process works and the randomness of Jenkins that it would be possible for someone who doesn't have a lot of experience troubleshooting maven errors (Something I've wrangling with years) would be able to solve easily.
> 
> On Fri, Jan 18, 2013 at 8:27 AM, Neal Caidin <nealcaidin at sakaifoundation.org> wrote:
> Yes, no new blockers regressions, and have a fairly high bar for a show stopper. I haven't checked this morning to see if they have been fixed/merged, but we do have a few security issues that possibly won't get done in time.
> 
> I'm at EuroSakai the week of Jan 28, that might make the coordination tricky. Unlike Educause where I was barely a blip, I expect I will be getting a lot of attention in Paris and will be hard pressed to find time for QA coordination/check-in. Would be nice to have a backup QA lead, but I don't.
> 
> So maybe the schedule range is more of the first or second week in February. But if we need an RC02, then that schedule fails, correct? And I need to have a QA lead/check in backup plan. If I don't have that, it could delay things a week and we are back to a mid-February.
> 
> - Neal
> 
> 
> 
> 
> _______________________________________________
> sakai2-tcc mailing list
> sakai2-tcc at collab.sakaiproject.org
> http://collab.sakaiproject.org/mailman/listinfo/sakai2-tcc

-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://collab.sakaiproject.org/pipermail/sakai2-tcc/attachments/20130119/c2c3d791/attachment-0001.html