[cle-release-team] An interesting (bug?) : release problem

Aaron Zeckoski azeckoski at unicon.net
Thu Jul 5 11:50:30 PDT 2012 

Well, I still think there is a slight "chicken and egg" (ordering)
issue here if we put the kernel versions in the master pom and then
the kernel inherits from that.... but I suppose that can be managed. I
would worry that it would confuse the release plugin and other plugins
that work like it (unless we stop using SNAPSHOT in kernel) but if it
works then I am all for it.

-AZ


On Thu, Jul 5, 2012 at 2:44 PM, Matthew Jones <matthew at longsight.com> wrote:
> I'm in agreement with Steve, I don't think the size of the DM block matters,
> and I'd rather have that DM block in a 'master' style pom than in the kernel
> pom, rather than what we have now with the kernel DM imported into the
> master causing a dependency on the kernel existing prior to releasing the
> master.
>
> I think this has to happen for 2.9.x and the build will either remain broken
> or 'unreliable' (meaning the artifacts it pulls down might not be the ones
> you customize) until it is fixed. I think we can still do releases fine
> (incorrectly) without doing this but if we don't do this we're going to have
> to tell people that if they want to do a local build they have to always
> build kernel separately before everything else, then they can build the
> rest, as 2 parts.
>
> So it's basically like:
> cd kernel && mvn clean install  # Build the kernel
> cd .. && mvn clean install   #Build everything else
>
> Otherwise local customized builds will either fail or grab the wrong kernel
> because it will take the one from the maven snapshots (as this is how
> snapshots work).
>
> On your other points David, I really don't believe pure poms simplified
> things or sped up releases. I'd ask Steve or Anthony since they were doing
> 2.8 releases how fast it was. The lack of 2.8 releases (only got to 2.8.2)
> mostly shows this. We should have had many more 2.8 releases. To actually
> get a release out, you had to modify properties of versions in master and
> properties of versions in 4 pure poms. You also had to modify tool poms
> individually to update properties. This wouldn't have been too bad if we had
> project teams working on these tools but generally it was just one person,
> and when Anthony left it was mostly volunteers (me and Steve).
>
> Then afterward to get the indies out, the order was a lot more important and
> needed to be determined manually, something Maven typically figures out as
> part of the reactor. There were a specific order of 24 steps verses the 8 or
> so now. (Which probably can even be less) Releasing indies was certainly
> non-trivial.
> https://confluence.sakaiproject.org/display/REL/Indie+release+order+-+pre+2.9
>
> And who really knew when to use the standard tool over the basic tool, where
> was this documented? And what do you do if you want to run a "old school
> style" monolithic build, eliminating core deploy, it generally involved you
> modifying a lot more files since there were at least 5 potential parents. I
> know because I spent a *lot* of time at Michigan trying to get this working
> correctly.
>
> Perhaps it was easier for some use case, but it didn't satisfy all use
> cases. This were a lot of the problems I was hearing on the list, on the
> calls and at the conference. Some wanted to get rid of indies entirely,
> mostly because it is just 1-2 people doing the release of 95% of the tools.
> Some wanted to keep some indies. I really just want it to work without
> running into a new problem every single release. ;)
>
> On Thu, Jul 5, 2012 at 7:05 AM, Steve Swinsburg <steve.swinsburg at gmail.com>
> wrote:
>>
>> I don't think it needs to be that complex.
>>
>> Kernel can just be an indie like any other. Master defines the versions of
>> many other indies as well so the kernel doesn't need to be anything special
>> in that regard. So get rid of master depending on kernel (via the import)
>> and have it the other way around where kernel just lists master as a parent.
>> Then you can have simply, for indies:
>>
>> sonatype -> master -> kernel/profile2/lessons/etc
>>
>> They can still depend on each other if they need to, e.g. profile2
>> depending on kernel or lessons depending on profile2.
>>
>> And for the other tools:
>>
>> sonatype -> master -> base -> alias/courier/help
>>
>> cheers,
>> S
>>
>> On 05/07/2012, at 8:55 PM, Aaron Zeckoski wrote:
>>
>> > I wouldn't worry too much about the size of the DM block. Lots of
>> > other projects have lists of dependencies that are miles long.
>> >
>> > I would worry about changing the way master works too much because
>> > most contrib tools depend on it directly so if we take too much out we
>> > will break all the contrib tools and that's not an acceptable fix for
>> > the community.
>> >
>> > I do like the idea of the kernel POM depending on the master POM but
>> > I'm not sure this makes sense since master defines the kernel versions
>> > in DM. I definitely don't like the way master imports the kernel pom
>> > the way it does. That has always bothered me.
>> >> From the master POM:
>> >                <dependency>
>> >                        <artifactId>kernel</artifactId>
>> >                        <groupId>org.sakaiproject</groupId>
>> >                        <version>${sakai.kernel.version}</version>
>> >                        <type>pom</type>
>> >                        <scope>import</scope>
>> >                </dependency>
>> > (and it depends on other parts of the kernel as well)
>> >
>> > I suspect we probably need something like:
>> > sonatype -> super-master -> kernel -> master -> tool-base or sakai-base
>> >
>> > (or something to that effect)
>> >
>> > -AZ
>> >
>> >
>> > On Thu, Jul 5, 2012 at 3:15 AM, David Horowitz <david.horwitz at uct.ac.za>
>> > wrote:
>> >> However any task that does dependency analysis will also analyze the
>> >> dm,
>> >> this is partly why we created the pure poms. It greatly speeded up
>> >> releases
>> >> by rationalizing the amount of artifacts scanned doing the analysis
>> >> phase.
>> >>
>> >> I'm also not sure why that very old junky is defined that high up ....
>> >>
>> >>
>> >> It feels to me here we are throwing out the baby with the bath water
>> >> and
>> >> ending up with a big monolithic , fragile build.
>> >>
>> >> D
>> >>
>> >> Sent from my iPad
>> >>
>> >> On 04 Jul 2012, at 7:51 PM, Matthew Jones <matthew at longsight.com>
>> >> wrote:
>> >>
>> >> The only dependency master has is junit. Everything else it just
>> >> dependency
>> >> management which every tool that inherits from it is free to use or
>> >> override
>> >> if they prefer to use a different version. It was seen more of a
>> >> convenience. By moving the kernel dependencyManagement stuff over into
>> >> master as well, it will add a lot more dependencyManagement, but all of
>> >> these are already "imported" anyway. Everything that has master as a
>> >> parent
>> >> already imports all of the kernel dependencyManagement because of
>> >> SAK-18921.
>> >>
>> >> I just think it would make sense having master on top of deverything.
>> >> Our
>> >> dependency tree should remain the same as no new dependencies are added
>> >> from
>> >> this that aren't already there. The only thing it would do is make it
>> >> easier
>> >> for doing releases since right now I have to release kernel manually,
>> >> then
>> >> release master manually, then I can release all of the indies
>> >> semi-automatically.
>> >>
>> >> If I do this, I can release master manually (or possibly automatically)
>> >> then
>> >> everything else semi-automatically, including kernel.
>> >>
>> >> Master version may go up high and fast, but I think keeping the 2.9,
>> >> 2.10 in
>> >> front is a still a good idea.
>> >>
>> >> If we actually had more people working on these 30 individual tools
>> >> rather
>> >> than just the 4-5 of us, then also moving the properties and
>> >> dependencyManagement out would make sense, but this just seems like the
>> >> only
>> >> way to manage a code-base this huge with ~2 days available a month for
>> >> each
>> >> person doing releases to work on it.
>> >>
>> >> On Wed, Jul 4, 2012 at 11:28 AM, David Horowitz
>> >> <david.horwitz at uct.ac.za>
>> >> wrote:
>> >>>
>> >>> Some things to consider
>> >>>
>> >>> 1 if there is 1 high level Pom it should be simple, it should not
>> >>> define
>> >>> every API in shared the way master Does. Otherwise all we are doing is
>> >>> breaking the work to clean up our dependency tree by declaring a
>> >>> dependency
>> >>> by everything on everything.
>> >>>
>> >>> There fore it should really just be framework dependencies, build
>> >>> profiles
>> >>> and licenses etc
>> >>>
>> >>> The scope of a maven property is the Pom it's declared in, really we
>> >>> should not be using properties in master anywhere else
>> >>>
>> >>> D
>> >>>
>> >>> Sent from my iPad
>> >>>
>> >>> On 04 Jul 2012, at 1:04 PM, Aaron Zeckoski <aaronz at vt.edu> wrote:
>> >>>
>> >>>>> I suppose this wasn't done because I believe there was some idea
>> >>>>> that
>> >>>>> kernels might release faster than the regular cycle of Sakai and
>> >>>>> master
>> >>>>> would get out of sync. And what version would you release master as?
>> >>>>> This
>> >>>>
>> >>>> Other way around. The idea was that kernel would not be changed as
>> >>>> often as the rest of Sakai and it would be released less.
>> >>>>
>> >>>> -AZ
>> >>>>
>> >>>>
>> >>>> On Wed, Jul 4, 2012 at 1:24 AM, Matthew Jones <matthew at longsight.com>
>> >>>> wrote:
>> >>>>> I agree, it kind of feels like at this point that master should be
>> >>>>> the
>> >>>>> parent for kernel, all of those dependencies defined in kernel that
>> >>>>> were
>> >>>>> imported in SAK-18921 moved into master, rather than the other way
>> >>>>> around.
>> >>>>>
>> >>>>> All master does is define properties and dependencyManagement. I
>> >>>>> believe if
>> >>>>> that was worked around so that we just made kernel a "regular indie"
>> >>>>> and it
>> >>>>> was
>> >>>>>
>> >>>>> oss-parent<-master<-all indies (including kernel)
>> >>>>> oss-parent<-master<-base<-all regular stuff
>> >>>>>
>> >>>>> I was kind of hoping at one time for some type of "indie-base" that
>> >>>>> maven
>> >>>>> could use to help figure out the release order so it didn't have to
>> >>>>> be
>> >>>>> manually defined. I'm not sure if this was possible or not.
>> >>>>>
>> >>>>> Perhaps this is the missing piece at the top?
>> >>>>>
>> >>>>> I suppose this wasn't done because I believe there was some idea
>> >>>>> that
>> >>>>> kernels might release faster than the regular cycle of Sakai and
>> >>>>> master
>> >>>>> would get out of sync. And what version would you release master as?
>> >>>>> This
>> >>>>> hasn't happened and 2.9 Sakai has always matched with 1.3.kernel.
>> >>>>> Potentially a new master for each indie release of anything, but
>> >>>>> they've
>> >>>>> been locked together since I've been doing it.
>> >>>>>
>> >>>>> Would we ever have a 2 *released* Sakai 2.9's, one with a 1.3 kernel
>> >>>>> and one
>> >>>>> with a 1.4 kernel? That really doesn't feel likely at all anymore. I
>> >>>>> think
>> >>>>> even if we did we'd still have the confusion in the present state.
>> >>>>>
>> >>>>> -Matthew
>> >>>>>
>> >>>>>
>> >>>>> On Tue, Jul 3, 2012 at 7:41 PM, Steve Swinsburg
>> >>>>> <steve.swinsburg at gmail.com>
>> >>>>> wrote:
>> >>>>>>
>> >>>>>> It seems odd that the kernel needs to be built before the master.
>> >>>>>> Master
>> >>>>>> should be the top level parent for everything.
>> >>>>>>
>> >>>>>> In https://jira.sakaiproject.org/browse/SAK-18921 the rationale was
>> >>>>>> that
>> >>>>>> these artifacts were deployed by the kernel so they need to be kept
>> >>>>>> in
>> >>>>>> sync
>> >>>>>> with both projects, therefore use the import dependency. I would
>> >>>>>> suggest
>> >>>>>> these be moved out so instead of being deployed by the kernel they
>> >>>>>> are
>> >>>>>> deployed by another pom, kernel-deploy would be a good spot.
>> >>>>>>
>> >>>>>> cheers,
>> >>>>>> Steve
>> >>>>>>
>> >>>>>>
>> >>>>>> On 04/07/2012, at 5:38 AM, Matthew Jones wrote:
>> >>>>>>
>> >>>>>> So the Sakai release/build process is pretty bad. If you build it
>> >>>>>> with
>> >>>>>> a
>> >>>>>> populated repository you get unpredictable results. If you build it
>> >>>>>> with an
>> >>>>>> empty repository it breaks. I'm not sure if this is a bug in maven
>> >>>>>> or
>> >>>>>> a bug
>> >>>>>> in Sakai at the moment.
>> >>>>>>
>> >>>>>> The problem with this really is this:
>> >>>>>> https://jira.sakaiproject.org/browse/SAK-18921
>> >>>>>>
>> >>>>>> Master has an import dependency on kernel so it requires kernel to
>> >>>>>> be
>> >>>>>> built before it can work. So I decided to try to change the master
>> >>>>>> parent to
>> >>>>>> kernel instead of sonatype so it would always build kernel first
>> >>>>>> correctly
>> >>>>>> as part of the reactor order (it still needs to build from base)
>> >>>>>> https://jira.sakaiproject.org/browse/SAK-22353
>> >>>>>>
>> >>>>>> So when you run "mvn clean install" on the base pom, it should see
>> >>>>>> it
>> >>>>>> has
>> >>>>>> a parent of master->has a parent of kernel and know it needs to
>> >>>>>> build
>> >>>>>> everything.
>> >>>>>>
>> >>>>>> It all seemed okay *except* when the build gets to the first tool
>> >>>>>> that
>> >>>>>> has
>> >>>>>> a kernel dependency, it wants the wrong kernel version!
>> >>>>>>
>> >>>>>>
>> >>>>>>
>> >>>>>> http://nightly2.sakaiproject.org/logs/sakai-rel/build-2012-07-03-01:00:02.log.txt
>> >>>>>>
>> >>>>>> 1) org.sakaiproject.kernel:sakai-component-manager:jar:1.2-SNAPSHOT
>> >>>>>>
>> >>>>>> This is defined in kernel pom's dependencyManagement with the
>> >>>>>> "${project.version}" variable. And 1.2-SNAPSHOT is the version of
>> >>>>>> common
>> >>>>>>
>> >>>>>> <dependency>
>> >>>>>> <groupId>org.sakaiproject.kernel</groupId>
>> >>>>>> <artifactId>sakai-component-manager</artifactId>
>> >>>>>> <version>${project.version}</version>
>> >>>>>> <scope>provided</scope>
>> >>>>>> </dependency>
>> >>>>>> <dependency>
>> >>>>>> <groupId>org.sakaiproject.kernel</groupId>
>> >>>>>> <artifactId>sakai-kernel-api</artifactId>
>> >>>>>> <version>${project.version}</version>
>> >>>>>> <scope>provided</scope>
>> >>>>>> </dependency>
>> >>>>>> <dependency>
>> >>>>>> <groupId>org.sakaiproject.kernel.component.api</groupId>
>> >>>>>> <artifactId>sakai-component-manager</artifactId>
>> >>>>>> <version>${project.version}</version>
>> >>>>>> <scope>provided</scope>
>> >>>>>> </dependency>
>> >>>>>> </dependencies>
>> >>>>>>
>> >>>>>> Somehow the version of the end child (common) is going all the way
>> >>>>>> up
>> >>>>>> to
>> >>>>>> chain to the parent, and it's using that rather than the
>> >>>>>> 1.3-SNAPSHOT
>> >>>>>> defined in that pom. If I change this ${project.version} to be
>> >>>>>> "1.3-SNAPSHOT" (or as a property with a value of 1.3-SNAPSHOT) in
>> >>>>>> these 3
>> >>>>>> places, the build works. If I revert SAK-18921 and put the parent
>> >>>>>> of
>> >>>>>> master
>> >>>>>> back to sonatype it works too. (It's able to figure out kernel
>> >>>>>> needs
>> >>>>>> to be
>> >>>>>> built first as it's a dependency for other projects, it just can't
>> >>>>>> figure it
>> >>>>>> out as an import dependency).
>> >>>>>>
>> >>>>>> Maven 3 is even worse, because it fails completely without the
>> >>>>>> relativePaths that were added in
>> >>>>>> https://jira.sakaiproject.org/browse/SAK-22321 .
>> >>>>>>
>> >>>>>> So I don't know what's up with this ${project.version}. Maybe it's
>> >>>>>> bugged
>> >>>>>> for multi module projects? I see a lot of random jiras. Same
>> >>>>>> behavior
>> >>>>>> in
>> >>>>>> Maven 3.0.4 and 2.2.1.
>> >>>>>>
>> >>>>>> I'm thinking of just moving the
>> >>>>>>
>> >>>>>>       <sakai.kernel.version>1.3.0-SNAPSHOT</sakai.kernel.version>
>> >>>>>>
>> >>>>>> property into kernel, and using that? I don't know . . .
>> >>>>>> _______________________________________________
>> >>>>>> cle-release-team mailing list
>> >>>>>> cle-release-team at collab.sakaiproject.org
>> >>>>>> http://collab.sakaiproject.org/mailman/listinfo/cle-release-team
>> >>>>>>
>> >>>>>>
>> >>>>>
>> >>>>>
>> >>>>> _______________________________________________
>> >>>>> cle-release-team mailing list
>> >>>>> cle-release-team at collab.sakaiproject.org
>> >>>>> http://collab.sakaiproject.org/mailman/listinfo/cle-release-team
>> >>>>>
>> >>>>
>> >>>>
>> >>>>
>> >>>> --
>> >>>> Aaron Zeckoski - Software Architect - http://tinyurl.com/azprofile
>> >>>> _______________________________________________
>> >>>> cle-release-team mailing list
>> >>>> cle-release-team at collab.sakaiproject.org
>> >>>> http://collab.sakaiproject.org/mailman/listinfo/cle-release-team
>> >>>>
>> >>
>> >>
>> >
>> >
>> >
>> > --
>> > Aaron Zeckoski - Software Architect - http://tinyurl.com/azprofile
>> > _______________________________________________
>> > cle-release-team mailing list
>> > cle-release-team at collab.sakaiproject.org
>> > http://collab.sakaiproject.org/mailman/listinfo/cle-release-team
>>
>> _______________________________________________
>> cle-release-team mailing list
>> cle-release-team at collab.sakaiproject.org
>> http://collab.sakaiproject.org/mailman/listinfo/cle-release-team
>
>
>
> _______________________________________________
> cle-release-team mailing list
> cle-release-team at collab.sakaiproject.org
> http://collab.sakaiproject.org/mailman/listinfo/cle-release-team
>



-- 
Aaron Zeckoski - Software Architect - http://tinyurl.com/azprofile

More information about the cle-release-team mailing list