[cle-release-team] An interesting (bug?) : release problem

Matthew Jones matthew at longsight.com
Thu Jul 5 11:44:46 PDT 2012 

I'm in agreement with Steve, I don't think the size of the DM block
matters, and I'd rather have that DM block in a 'master' style pom than in
the kernel pom, rather than what we have now with the kernel DM imported
into the master causing a dependency on the kernel existing prior to
releasing the master.

I think this has to happen for 2.9.x and the build will either remain
broken or 'unreliable' (meaning the artifacts it pulls down might not be
the ones you customize) until it is fixed. I think we can still do releases
fine (incorrectly) without doing this but if we don't do this we're going
to have to tell people that if they want to do a local build they have to
always build kernel separately before everything else, then they can build
the rest, as 2 parts.

So it's basically like:
cd kernel && mvn clean install  # Build the kernel
cd .. && mvn clean install   #Build everything else

Otherwise local customized builds will either fail or grab the wrong kernel
because it will take the one from the maven snapshots (as this is how
snapshots work).

On your other points David, I really don't believe pure poms simplified
things or sped up releases. I'd ask Steve or Anthony since they were doing
2.8 releases how fast it was. The lack of 2.8 releases (only got to 2.8.2)
mostly shows this. We should have had many more 2.8 releases. To actually
get a release out, you had to modify properties of versions in master and
properties of versions in 4 pure poms. You also had to modify tool poms
individually to update properties. This wouldn't have been too bad if we
had project teams working on these tools but generally it was just one
person, and when Anthony left it was mostly volunteers (me and Steve).

Then afterward to get the indies out, the order was a lot more important
and needed to be determined manually, something Maven typically figures out
as part of the reactor. There were a specific order of 24 steps verses the
8 or so now. (Which probably can even be less) Releasing indies was
certainly non-trivial.
https://confluence.sakaiproject.org/display/REL/Indie+release+order+-+pre+2.9

And who really knew when to use the standard tool over the basic tool,
where was this documented? And what do you do if you want to run a "old
school style" monolithic build, eliminating core deploy, it generally
involved you modifying a lot more files since there were at least 5
potential parents. I know because I spent a *lot* of time at Michigan
trying to get this working correctly.

Perhaps it was easier for some use case, but it didn't satisfy all use
cases. This were a lot of the problems I was hearing on the list, on the
calls and at the conference. Some wanted to get rid of indies entirely,
mostly because it is just 1-2 people doing the release of 95% of the tools.
Some wanted to keep some indies. I really just want it to work without
running into a new problem every single release. ;)

On Thu, Jul 5, 2012 at 7:05 AM, Steve Swinsburg
<steve.swinsburg at gmail.com>wrote:

> I don't think it needs to be that complex.
>
> Kernel can just be an indie like any other. Master defines the versions of
> many other indies as well so the kernel doesn't need to be anything special
> in that regard. So get rid of master depending on kernel (via the import)
> and have it the other way around where kernel just lists master as a
> parent. Then you can have simply, for indies:
>
> sonatype -> master -> kernel/profile2/lessons/etc
>
> They can still depend on each other if they need to, e.g. profile2
> depending on kernel or lessons depending on profile2.
>
> And for the other tools:
>
> sonatype -> master -> base -> alias/courier/help
>
> cheers,
> S
>
> On 05/07/2012, at 8:55 PM, Aaron Zeckoski wrote:
>
> > I wouldn't worry too much about the size of the DM block. Lots of
> > other projects have lists of dependencies that are miles long.
> >
> > I would worry about changing the way master works too much because
> > most contrib tools depend on it directly so if we take too much out we
> > will break all the contrib tools and that's not an acceptable fix for
> > the community.
> >
> > I do like the idea of the kernel POM depending on the master POM but
> > I'm not sure this makes sense since master defines the kernel versions
> > in DM. I definitely don't like the way master imports the kernel pom
> > the way it does. That has always bothered me.
> >> From the master POM:
> >                <dependency>
> >                        <artifactId>kernel</artifactId>
> >                        <groupId>org.sakaiproject</groupId>
> >                        <version>${sakai.kernel.version}</version>
> >                        <type>pom</type>
> >                        <scope>import</scope>
> >                </dependency>
> > (and it depends on other parts of the kernel as well)
> >
> > I suspect we probably need something like:
> > sonatype -> super-master -> kernel -> master -> tool-base or sakai-base
> >
> > (or something to that effect)
> >
> > -AZ
> >
> >
> > On Thu, Jul 5, 2012 at 3:15 AM, David Horowitz <david.horwitz at uct.ac.za>
> wrote:
> >> However any task that does dependency analysis will also analyze the dm,
> >> this is partly why we created the pure poms. It greatly speeded up
> releases
> >> by rationalizing the amount of artifacts scanned doing the analysis
> phase.
> >>
> >> I'm also not sure why that very old junky is defined that high up ....
> >>
> >>
> >> It feels to me here we are throwing out the baby with the bath water and
> >> ending up with a big monolithic , fragile build.
> >>
> >> D
> >>
> >> Sent from my iPad
> >>
> >> On 04 Jul 2012, at 7:51 PM, Matthew Jones <matthew at longsight.com>
> wrote:
> >>
> >> The only dependency master has is junit. Everything else it just
> dependency
> >> management which every tool that inherits from it is free to use or
> override
> >> if they prefer to use a different version. It was seen more of a
> >> convenience. By moving the kernel dependencyManagement stuff over into
> >> master as well, it will add a lot more dependencyManagement, but all of
> >> these are already "imported" anyway. Everything that has master as a
> parent
> >> already imports all of the kernel dependencyManagement because of
> SAK-18921.
> >>
> >> I just think it would make sense having master on top of deverything.
> Our
> >> dependency tree should remain the same as no new dependencies are added
> from
> >> this that aren't already there. The only thing it would do is make it
> easier
> >> for doing releases since right now I have to release kernel manually,
> then
> >> release master manually, then I can release all of the indies
> >> semi-automatically.
> >>
> >> If I do this, I can release master manually (or possibly automatically)
> then
> >> everything else semi-automatically, including kernel.
> >>
> >> Master version may go up high and fast, but I think keeping the 2.9,
> 2.10 in
> >> front is a still a good idea.
> >>
> >> If we actually had more people working on these 30 individual tools
> rather
> >> than just the 4-5 of us, then also moving the properties and
> >> dependencyManagement out would make sense, but this just seems like the
> only
> >> way to manage a code-base this huge with ~2 days available a month for
> each
> >> person doing releases to work on it.
> >>
> >> On Wed, Jul 4, 2012 at 11:28 AM, David Horowitz <
> david.horwitz at uct.ac.za>
> >> wrote:
> >>>
> >>> Some things to consider
> >>>
> >>> 1 if there is 1 high level Pom it should be simple, it should not
> define
> >>> every API in shared the way master Does. Otherwise all we are doing is
> >>> breaking the work to clean up our dependency tree by declaring a
> dependency
> >>> by everything on everything.
> >>>
> >>> There fore it should really just be framework dependencies, build
> profiles
> >>> and licenses etc
> >>>
> >>> The scope of a maven property is the Pom it's declared in, really we
> >>> should not be using properties in master anywhere else
> >>>
> >>> D
> >>>
> >>> Sent from my iPad
> >>>
> >>> On 04 Jul 2012, at 1:04 PM, Aaron Zeckoski <aaronz at vt.edu> wrote:
> >>>
> >>>>> I suppose this wasn't done because I believe there was some idea that
> >>>>> kernels might release faster than the regular cycle of Sakai and
> master
> >>>>> would get out of sync. And what version would you release master as?
> >>>>> This
> >>>>
> >>>> Other way around. The idea was that kernel would not be changed as
> >>>> often as the rest of Sakai and it would be released less.
> >>>>
> >>>> -AZ
> >>>>
> >>>>
> >>>> On Wed, Jul 4, 2012 at 1:24 AM, Matthew Jones <matthew at longsight.com>
> >>>> wrote:
> >>>>> I agree, it kind of feels like at this point that master should be
> the
> >>>>> parent for kernel, all of those dependencies defined in kernel that
> >>>>> were
> >>>>> imported in SAK-18921 moved into master, rather than the other way
> >>>>> around.
> >>>>>
> >>>>> All master does is define properties and dependencyManagement. I
> >>>>> believe if
> >>>>> that was worked around so that we just made kernel a "regular indie"
> >>>>> and it
> >>>>> was
> >>>>>
> >>>>> oss-parent<-master<-all indies (including kernel)
> >>>>> oss-parent<-master<-base<-all regular stuff
> >>>>>
> >>>>> I was kind of hoping at one time for some type of "indie-base" that
> >>>>> maven
> >>>>> could use to help figure out the release order so it didn't have to
> be
> >>>>> manually defined. I'm not sure if this was possible or not.
> >>>>>
> >>>>> Perhaps this is the missing piece at the top?
> >>>>>
> >>>>> I suppose this wasn't done because I believe there was some idea that
> >>>>> kernels might release faster than the regular cycle of Sakai and
> master
> >>>>> would get out of sync. And what version would you release master as?
> >>>>> This
> >>>>> hasn't happened and 2.9 Sakai has always matched with 1.3.kernel.
> >>>>> Potentially a new master for each indie release of anything, but
> >>>>> they've
> >>>>> been locked together since I've been doing it.
> >>>>>
> >>>>> Would we ever have a 2 *released* Sakai 2.9's, one with a 1.3 kernel
> >>>>> and one
> >>>>> with a 1.4 kernel? That really doesn't feel likely at all anymore. I
> >>>>> think
> >>>>> even if we did we'd still have the confusion in the present state.
> >>>>>
> >>>>> -Matthew
> >>>>>
> >>>>>
> >>>>> On Tue, Jul 3, 2012 at 7:41 PM, Steve Swinsburg
> >>>>> <steve.swinsburg at gmail.com>
> >>>>> wrote:
> >>>>>>
> >>>>>> It seems odd that the kernel needs to be built before the master.
> >>>>>> Master
> >>>>>> should be the top level parent for everything.
> >>>>>>
> >>>>>> In https://jira.sakaiproject.org/browse/SAK-18921 the rationale was
> >>>>>> that
> >>>>>> these artifacts were deployed by the kernel so they need to be kept
> in
> >>>>>> sync
> >>>>>> with both projects, therefore use the import dependency. I would
> >>>>>> suggest
> >>>>>> these be moved out so instead of being deployed by the kernel they
> are
> >>>>>> deployed by another pom, kernel-deploy would be a good spot.
> >>>>>>
> >>>>>> cheers,
> >>>>>> Steve
> >>>>>>
> >>>>>>
> >>>>>> On 04/07/2012, at 5:38 AM, Matthew Jones wrote:
> >>>>>>
> >>>>>> So the Sakai release/build process is pretty bad. If you build it
> with
> >>>>>> a
> >>>>>> populated repository you get unpredictable results. If you build it
> >>>>>> with an
> >>>>>> empty repository it breaks. I'm not sure if this is a bug in maven
> or
> >>>>>> a bug
> >>>>>> in Sakai at the moment.
> >>>>>>
> >>>>>> The problem with this really is this:
> >>>>>> https://jira.sakaiproject.org/browse/SAK-18921
> >>>>>>
> >>>>>> Master has an import dependency on kernel so it requires kernel to
> be
> >>>>>> built before it can work. So I decided to try to change the master
> >>>>>> parent to
> >>>>>> kernel instead of sonatype so it would always build kernel first
> >>>>>> correctly
> >>>>>> as part of the reactor order (it still needs to build from base)
> >>>>>> https://jira.sakaiproject.org/browse/SAK-22353
> >>>>>>
> >>>>>> So when you run "mvn clean install" on the base pom, it should see
> it
> >>>>>> has
> >>>>>> a parent of master->has a parent of kernel and know it needs to
> build
> >>>>>> everything.
> >>>>>>
> >>>>>> It all seemed okay *except* when the build gets to the first tool
> that
> >>>>>> has
> >>>>>> a kernel dependency, it wants the wrong kernel version!
> >>>>>>
> >>>>>>
> >>>>>>
> http://nightly2.sakaiproject.org/logs/sakai-rel/build-2012-07-03-01:00:02.log.txt
> >>>>>>
> >>>>>> 1) org.sakaiproject.kernel:sakai-component-manager:jar:1.2-SNAPSHOT
> >>>>>>
> >>>>>> This is defined in kernel pom's dependencyManagement with the
> >>>>>> "${project.version}" variable. And 1.2-SNAPSHOT is the version of
> >>>>>> common
> >>>>>>
> >>>>>> <dependency>
> >>>>>> <groupId>org.sakaiproject.kernel</groupId>
> >>>>>> <artifactId>sakai-component-manager</artifactId>
> >>>>>> <version>${project.version}</version>
> >>>>>> <scope>provided</scope>
> >>>>>> </dependency>
> >>>>>> <dependency>
> >>>>>> <groupId>org.sakaiproject.kernel</groupId>
> >>>>>> <artifactId>sakai-kernel-api</artifactId>
> >>>>>> <version>${project.version}</version>
> >>>>>> <scope>provided</scope>
> >>>>>> </dependency>
> >>>>>> <dependency>
> >>>>>> <groupId>org.sakaiproject.kernel.component.api</groupId>
> >>>>>> <artifactId>sakai-component-manager</artifactId>
> >>>>>> <version>${project.version}</version>
> >>>>>> <scope>provided</scope>
> >>>>>> </dependency>
> >>>>>> </dependencies>
> >>>>>>
> >>>>>> Somehow the version of the end child (common) is going all the way
> up
> >>>>>> to
> >>>>>> chain to the parent, and it's using that rather than the
> 1.3-SNAPSHOT
> >>>>>> defined in that pom. If I change this ${project.version} to be
> >>>>>> "1.3-SNAPSHOT" (or as a property with a value of 1.3-SNAPSHOT) in
> >>>>>> these 3
> >>>>>> places, the build works. If I revert SAK-18921 and put the parent of
> >>>>>> master
> >>>>>> back to sonatype it works too. (It's able to figure out kernel needs
> >>>>>> to be
> >>>>>> built first as it's a dependency for other projects, it just can't
> >>>>>> figure it
> >>>>>> out as an import dependency).
> >>>>>>
> >>>>>> Maven 3 is even worse, because it fails completely without the
> >>>>>> relativePaths that were added in
> >>>>>> https://jira.sakaiproject.org/browse/SAK-22321 .
> >>>>>>
> >>>>>> So I don't know what's up with this ${project.version}. Maybe it's
> >>>>>> bugged
> >>>>>> for multi module projects? I see a lot of random jiras. Same
> behavior
> >>>>>> in
> >>>>>> Maven 3.0.4 and 2.2.1.
> >>>>>>
> >>>>>> I'm thinking of just moving the
> >>>>>>
> >>>>>>       <sakai.kernel.version>1.3.0-SNAPSHOT</sakai.kernel.version>
> >>>>>>
> >>>>>> property into kernel, and using that? I don't know . . .
> >>>>>> _______________________________________________
> >>>>>> cle-release-team mailing list
> >>>>>> cle-release-team at collab.sakaiproject.org
> >>>>>> http://collab.sakaiproject.org/mailman/listinfo/cle-release-team
> >>>>>>
> >>>>>>
> >>>>>
> >>>>>
> >>>>> _______________________________________________
> >>>>> cle-release-team mailing list
> >>>>> cle-release-team at collab.sakaiproject.org
> >>>>> http://collab.sakaiproject.org/mailman/listinfo/cle-release-team
> >>>>>
> >>>>
> >>>>
> >>>>
> >>>> --
> >>>> Aaron Zeckoski - Software Architect - http://tinyurl.com/azprofile
> >>>> _______________________________________________
> >>>> cle-release-team mailing list
> >>>> cle-release-team at collab.sakaiproject.org
> >>>> http://collab.sakaiproject.org/mailman/listinfo/cle-release-team
> >>>>
> >>
> >>
> >
> >
> >
> > --
> > Aaron Zeckoski - Software Architect - http://tinyurl.com/azprofile
> > _______________________________________________
> > cle-release-team mailing list
> > cle-release-team at collab.sakaiproject.org
> > http://collab.sakaiproject.org/mailman/listinfo/cle-release-team
>
> _______________________________________________
> cle-release-team mailing list
> cle-release-team at collab.sakaiproject.org
> http://collab.sakaiproject.org/mailman/listinfo/cle-release-team
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://collab.sakaiproject.org/pipermail/cle-release-team/attachments/20120705/65d9c05a/attachment-0006.html

More information about the cle-release-team mailing list