[Building Sakai] How to checkout the 2.9.0 tag source including all tagged indie source?

[Cris J Holdorph]

I had a similar experience to Aaron, but with slightly tweaked code.  I 
worked on it a while.  I didn't exactly give up, but the situation I 
finally ended up in was a bunch of custom 1-offs that I had to build 
locally like of specific versions of purepom RELEASES, plugins, 
sakai-rsf, etc.  Once I had done all of those *AND* had updated a few of 
the sakai 2.9.x-all sources to point to my 1-offs, then I could finally 
remove all of the ~/.m2/repository/org/sakaiproject/ directory tree and 
do an offline build.  It did require doing things in the right order 
though (like plugin, purepoms, sakai-rsf, etc.).

---- Cris J H

On 11/28/2012 01:16 PM, Aaron Zeckoski wrote:
> I think that it *should* be possible to do an offline build once an
> "online" build has been completed successfully. However, if I delete
> the Sakai artifacts from my M2 repo and leave all the others I cannot
> do an offline build of the 2.9 code.
>
> I haven't had much luck convincing others that this is a real issue
> for community users but hopefully this conversation will help with
> that.
>
> -AZ
>
>
> On Wed, Nov 28, 2012 at 3:07 PM, Matthew Jones <matthew at longsight.com> wrote:
>> I believe what you were describing was being seen by Unicon, which was why I
>> had to keep cleaning the SNAPSHOTS from the repository and fixing bugs
>> awhile ago.
>>
>> Indies have a parent of master
>> The core has a parent of base which has a parent of master
>>
>> I believe that if you just dump the indies into base (which is not a parent
>> of any of the indies) then when maven builds up it's dependency tree, it
>> doesn't necessarily see that assignment is a dependency of lessons (for
>> example) since it can just go out to the repository and get all of the
>> artifacts that lessons needs. This is *probably* okay for the most part
>> since the dependencies are really just apis and most people are just
>> modifying the impls. But if you're modifying impls I don't think that maven
>> would necessarily order assignments (and everything else) before lessons
>> unless the parent of lessons was base. If you dumped the indies modules into
>> master, I'd probably be more confident.
>>
>> It's likely this didn't turn up in 2.7 because the only indies then were
>> kernel and like 2-3 others. There were some in 2.8 but less and most of the
>> time people were probably messed up (or partially saved) anyway with
>> purepoms and getting the versions that were defined locally in the top level
>> of each tool anyway. I don't know because Michigan never went to 2.8 and
>> Longsight has built off 2.8.x and also deploys tools individually.
>>
>> 2.9.0 (and maybe even the 2.9.x-all as far as indies are concerned) could
>> potentially cause all new problems unless it's 100% guaranteed that maven
>> does correct reactor ordering and never goes out to the repository for
>> artifacts that it has available to build. Since it's not possibly to run in
>> offline mode (because of artifacts that we don't have the source for) I'm
>> not too sure I'd 100% believe it without seeing a few build logs for myself.
>>
>>
>> On Wed, Nov 28, 2012 at 2:50 PM, Steve Swinsburg <steve.swinsburg at gmail.com>
>> wrote:
>>>
>>> Changing the version numbers should not be required. If the source module
>>> are listed in the base pom and there are no assembly deploy poms, then maven
>>> will build a dependency graph and adjust the build order to suit. It does
>>> that now, it's just that since there is no indie source it will get
>>> artifacts solely from the remote repo. If you look at the build you'll see
>>> individual jars being downloaded, like API etc, as well as the all in one
>>> tomcat overlay. It's redundant but that's how Maven sees the dependency
>>> graph.
>>>
>>> So adding the source modules as per the current -all branches, you should
>>> not get any sakai app artifacts being downloaded, they should all be built.
>>>
>>> If this does still occur then there is a fundamental issue, and it would
>>> have been seen in 2.7 and earlier, since as part of the tag/release process
>>> ALL artifacts are deployed to the Sakai maven repo and you would have seen
>>> the conflicts then.
>>>
>>> http://source.sakaiproject.org/maven2/org/sakaiproject/sakai-announcement-api/
>>>
>>> I'll have a look at the parts you posted Cynthia, and get this finalised
>>> today.
>>>
>>> Cheers
>>> Steve
>>>
>>> Sent from my iPhone
>>>
>>> On 29/11/2012, at 4:46, "Gast, Cynthia (cmw6s)"
>>> <cmw6s at eservices.virginia.edu> wrote:
>>>
>>> Matthew - wow, so much to know about building with all the indie source --
>>> thanks for sharing your knowledge.  I'm definitely still learning; I also
>>> wish it were easy to build everything without having to make pom version
>>> changes to avoid downloading community-built artifacts sneaking in.
>>> Customizing all the version numbers in poms takes a chunk of start-up time,
>>> and I found I really had to manually examine each pom for possible
>>> dependencies that contain hard-coded version numbers for projects we build;
>>> I didn't see how I could automate this sweeping version edit.  I'll look at
>>> the 'groovy' work you did.  Maybe 2.9.x will be better without purepoms; and
>>> some of this hard-coding of version numbers occurs in contrib tools.  The
>>> use of version properties, and defining a version number in one place, is
>>> extremely helpful, but I wasn't able to assume before that it was a practice
>>> that was consistently followed.
>>>
>>> So, for the first build of a new release (after making the uva-version
>>> updates), I closely examine the build logs for incorrect (not uva-built)
>>> downloaded versions of artifacts and adjust until those we build are
>>> referenced.
>>>
>>> We were considering not customizing the version numbers this time around,
>>> to save some start up time, but I'm glad we've had this discussion today
>>> because it now appears important that we continue to do so.
>>>
>>> Thanks,
>>> Cindy @ UVa
>>>
>>>
>>> ________________________________
>>> From: Matthew Jones [matthew at longsight.com]
>>> Sent: Wednesday, November 28, 2012 11:31 AM
>>> To: Gast, Cynthia (cmw6s)
>>> Cc: Steve Swinsburg; sakai-dev at collab.sakaiproject.org
>>> Subject: Re: [Building Sakai] How to checkout the 2.9.0 tag source
>>> including all tagged indie source?
>>>
>>> The "deploy" modules are not externals, they are actually *in* the
>>> directory, see
>>> https://source.sakaiproject.org/svn/sakai/tags/sakai-2.9.0/
>>>
>>> there is
>>> core-deploy/
>>> framework-shared-deploy/
>>> jstl-shared-deploy/
>>> kernel-deploy/
>>>
>>> In the 2.9.x-all there is only deploy (which deploys a few non-sakai
>>> things into common and shared) and jstl-shared-deploy (which deploys a few
>>> jstl things)
>>>
>>> With these 3 extra directory, the sakai:deploy could download the
>>> assemblies in the repository and use those as part of the build. You'll
>>> either need to remove those from the base pom or remove specific pieces you
>>> don't want from their individual poms. It's possible that it could grab
>>> assemblies that you build from your local repository but it still might
>>> check for updates on Sonatype and use those instead. (Unless you have
>>> different versions built and defined in master, then it would be forced to
>>> use those of course)
>>>
>>>
>>>
>>>
>>> On Wed, Nov 28, 2012 at 11:25 AM, Gast, Cynthia (cmw6s)
>>> <cmw6s at eservices.virginia.edu> wrote:
>>>>
>>>> Hello Steve and Matthew:
>>>> I'm attaching a file for your review, hopefully it will allow for the
>>>> checkout of all source for the 2.9.0 tag release, including the source for
>>>> all indie projects.
>>>>
>>>> It is a combination of the current 2.9.0-tag externals, plus all the
>>>> indies 'tag' releases as also represented in the externals file from
>>>> Matthew.
>>>>
>>>> I don't see the various "*-deploy" projects specifically listed, so I'm
>>>> assuming these will come along as they normally do with the 2.9.0-tag
>>>> checkout?
>>>>
>>>> I'm looking at this as a way to checkout (or export) the source for all
>>>> projects in the release.  Building same will be the next step :-)
>>>>
>>>> Thanks for your willingness to help with this - I'd love to have a
>>>> one-step source checkout.  Let me know when I should attempt a checkout for
>>>> all tag release source.
>>>>
>>>> Cheers,
>>>> Cindy @ UVa
>>>>
>>>>
>>>> ________________________________
>>>> From: sakai-dev-bounces at collab.sakaiproject.org
>>>> [sakai-dev-bounces at collab.sakaiproject.org] on behalf of Gast, Cynthia
>>>> (cmw6s) [cmw6s at eservices.virginia.edu]
>>>> Sent: Wednesday, November 28, 2012 8:53 AM
>>>> To: Steve Swinsburg; Matthew Jones
>>>> Cc: sakai-dev at collab.sakaiproject.org
>>>>
>>>> Subject: Re: [Building Sakai] How to checkout the 2.9.0 tag source
>>>> including all tagged indie source?
>>>>
>>>> Thanks Steve!
>>>> I'll make a stab at the externals this morning, send it to you both --
>>>> let you put it in place and I'll be happy to try it once you say so.
>>>>
>>>> Cindy
>>>>
>>>> ________________________________
>>>> From: Steve Swinsburg [steve.swinsburg at gmail.com]
>>>> Sent: Wednesday, November 28, 2012 6:40 AM
>>>> To: Matthew Jones
>>>> Cc: Gast, Cynthia (cmw6s); sakai-dev at collab.sakaiproject.org
>>>> Subject: Re: [Building Sakai] How to checkout the 2.9.0 tag source
>>>> including all tagged indie source?
>>>>
>>>> It should be pretty easy to just create a .externals that points to the
>>>> various tags instead of the branch, using the tagged 2.9.0 one as a base,
>>>> then adding in the indie tags. Then combine that with the tagged version of
>>>> core-deploy and the main pom and its done.
>>>>
>>>> I'd imagine that is what Chris was suggesting, rather than specific
>>>> revisions from the branch.
>>>> If it wasn't late in the eve I'd do it. Chris/Cynthia if you dont get to
>>>> it before Friday, I'll get onto this.
>>>>
>>>> cheers,
>>>> Steve
>>>>
>>>>
>>>> On Wed, Nov 28, 2012 at 12:19 PM, Matthew Jones <matthew at longsight.com>
>>>> wrote:
>>>>>
>>>>> I'd just use the 2.9.x-all branch. If you used 2.9.0 you'd have to mess
>>>>> around too much with the core-deploy and change too many other versions.
>>>>>
>>>>> If you wanted to lock it down to specific revisions, you'd want to have
>>>>> an msub in svn and make a copy of the 2.9.x-all branch, then either manually
>>>>> or with some script go down each external and set it to the revision which
>>>>> was tagged for 2.9.0
>>>>>
>>>>> #Pseudocode
>>>>> foreach external in 2.9.x-all {
>>>>> do a regex of 2.9.x to 2.9.0
>>>>> svn info on the 2.9.0 revision
>>>>> print back revision number in the external
>>>>> }
>>>>> svn propset external
>>>>> svn commit
>>>>>
>>>>> I had a script that *kind of* did this for Michigan, but it set the
>>>>> latest 2.9.x revision, and I had to manually bump it back if we didn't want
>>>>> that version.
>>>>>
>>>>> https://source.sakaiproject.org/svn/msub/umich.edu/ctools/builds/trunk/tools/versionSetter.pl
>>>>>
>>>>> I didn't think it would take more than an hour to write this script, it
>>>>> ended up taking a little more than 2 hours because the poms are pretty
>>>>> inconsistent. The names of the artifactId don't always match the names of
>>>>> the directory they're in, not all poms have version tags in them (so they
>>>>> inherit from the master, this is mostly true for the ones in core) and the
>>>>> core is still on a 2 digit version.
>>>>>
>>>>> However the versions for core and indies for 2.9.0 should be correct in
>>>>> this externals
>>>>> https://source.sakaiproject.org/contrib/cle-release/scripts/externals/
>>>>>
>>>>> Then if you did want some newer version you could just update it.
>>>>>
>>>>> What was your suggestion, Chris?
>>>>>
>>>>>
>>>>> On Tue, Nov 27, 2012 at 2:45 PM, Gast, Cynthia (cmw6s)
>>>>> <cmw6s at eservices.virginia.edu> wrote:
>>>>>>
>>>>>> Hello:
>>>>>>
>>>>>> We are setting up for our 2.9.0 upgrade.  We need source for all the
>>>>>> projects.  First, I exported the 2.9.0 tag, but this only provides source
>>>>>> for some projects (as has been discussed); all indie project tagged source
>>>>>> would have to be acquired still.  I took a look at the checkout using the
>>>>>> 2.9.x-all branches (svn export
>>>>>> https://source.sakaiproject.org/svn/sakai/branches/sakai-2.9.x-all/
>>>>>> ./2.9.x-all).  I see the externals is set to the trunk branches.  So while
>>>>>> it does allow me to acquire source for all the core + indie projects, it is
>>>>>> source from trunk (with pom files referring to SNAPSHOT versions).
>>>>>>
>>>>>> Has anyone created an externals that allows one to export the source
>>>>>> for the 2.9.0 tag plus the tag versions of the indies that make it up? ... a
>>>>>> 2.9.0-all ?  This would be a huge help to those of us who do build all
>>>>>> source.
>>>>>>
>>>>>> If not, would it help if I created the externals file to allow us to do
>>>>>> this?  Otherwise, I'll go get each indie's source one-by-one according to
>>>>>> the version tag.
>>>>>>
>>>>>> Thanks for considering this,
>>>>>> Cynthia Gast
>>>>>> UVa Sakai development team
>>>>>> cmw6s at virginia.edu
>>>>>>
>>>>>> _______________________________________________
>>>>>> sakai-dev mailing list
>>>>>> sakai-dev at collab.sakaiproject.org
>>>>>> http://collab.sakaiproject.org/mailman/listinfo/sakai-dev
>>>>>>
>>>>>> TO UNSUBSCRIBE: send email to
>>>>>> sakai-dev-unsubscribe at collab.sakaiproject.org with a subject of
>>>>>> "unsubscribe"
>>>>>
>>>>>
>>>>>
>>>>> _______________________________________________
>>>>> sakai-dev mailing list
>>>>> sakai-dev at collab.sakaiproject.org
>>>>> http://collab.sakaiproject.org/mailman/listinfo/sakai-dev
>>>>>
>>>>> TO UNSUBSCRIBE: send email to
>>>>> sakai-dev-unsubscribe at collab.sakaiproject.org with a subject of
>>>>> "unsubscribe"
>>>>
>>>>
>>>
>>
>>
>> _______________________________________________
>> sakai-dev mailing list
>> sakai-dev at collab.sakaiproject.org
>> http://collab.sakaiproject.org/mailman/listinfo/sakai-dev
>>
>> TO UNSUBSCRIBE: send email to sakai-dev-unsubscribe at collab.sakaiproject.org
>> with a subject of "unsubscribe"
>
>
>