[cle-release-team] [Building Sakai] Important information for trunk developers (SNAPSHOT artifacts)

Tue May 1 16:34:57 PDT 2012

There is doco on how to customise indies here:
https://confluence.sakaiproject.org/display/SAKDEV/Customizing+'indie'+releases

It could do with being less verbose and being condensed to a few simple steps.

Regarding scpexe, nothing negative, but since we could only deploy to the Sakai Maven Repo via SCP, then it was a change that had to be made, which affected all projects. And that kicked off the discussion about there having to be a better way. Hence the Sonatype move.

I agree that not everything should be indies, there should be a core set of apps that are in the build that are not indies. eg, why is portal an indie? You kind of really need that. So maybe a bit of rationalisation is in order, perhaps identifying the modules that are core, and then optional tools and services that if are not present the server won't explode, being set as indies.

I just things need to be simpler. Its easy for us as we are immersed in it and know all of the tricks, but take a step back and look at it from a newbies perspective, is it easy? (rhetorical q)

cheers,
Steve

On 01/05/2012, at 11:15 AM, Matthew Jones wrote:

> I have a feeling that the real problem was that there were a few projects (search, basiclti) where the poms were messed up, and it was only (obviously) uncovered after removing all of the snapshot artifacts. Aaron has been commenting about it for awhile but I was never able to reproduce his problem for whatever reason. As soon as all of the artifacts were removed, I was able to find and fix a number of issues with the build.
> 
> Search was a problem because the assembly had dependencies on test artifacts which were not built when skip.test=true (because of a profile in the root pom for search). I think this more showed which developers were skipping tests verses which developers actually had problems though. ;)
> 
> BasicLTI had a problem because the release plugin bumped the version (to 2.1-SNAPSHOT), which wasn't correctly updated in master. Since the (old) snapshot artifacts (for 2.0-SNAPSHOT) were in the repository, it was still using this version for something that depended on it. So everyone that was pulling BasicLTI in a regular trunk build was pulling the wrong version for the last 5 months.
> 
> I agree requiring a "-all" build goes against the point of indies, but there was some disagreement about if every tool that currently *is* indie really should be an indie, if it makes sense both as far as the amount of time it takes one person to release, manage and track everything everything everything. This is balanced for all of the institutions that want to feature patch indies. They need to either checkout the "-all" or do some process that we don't have documented. We don't have a document (AFIAK) that says:
> 
> - If you as an institution want to make local patches (pull in features specifically) against a tool that is an indie (downloaded by core-deploy) what should you do?
> - How about if you want to pull in local patches against a library? (Like jsf? edu-services? kernel?)
> - How are you sure your changes *are* in there for sure and maven didn't download artifacts/assemblies? (Other than parsing the build logs all the time or running in --offline mode?)
> 
> These were questions I wasn't ready to answer last Thursday, at least for trunk. These were all issues I wanted to either put off till the conference or at least try removing all artifacts to "see what happened". Generally it seems like if nothing "seems" to be broken than there's no discussion. As soon as it's broke, then people come out of nowhere. ;)
> 
> All 2.9 indies SNAPSHOTS and release *are* in sonatype. The decision to remove purepoms retroactively in 2.9 rather than waiting until 2.10 seemed like a good one. Purepoms only served to complicate the build order, complicated having to redefine and keep properties up to date and only a handful of people knew which purepom they were actually supposed to use. Though Seth suggested keeping them at least around as SNAPSHOTS (which I did), but it doesn't make any sense to release them, since nobody will reference released versions.
> 
> It's much easier just using master, it's essentially the same, it just has more properties and dependencyManagement defined. If you don't like the properties and dependencyManagement in master, just override it locally. But it's should be a "good default".
>  
> I'm not sure if there was a significant concern about the lack of server resources, but just strong recommendations from sonatype docs that all artifacts should just be in maven central (since this is in the master maven pom forever) [1] Since we were already deploying some things there, it seemed easier to just deploy everything there. This was a decision made on the release call. back in February. 
> 
> Which part of it do you want more streamlined? Jenkins, when you click "Perform Release" and fill in the info, release the artifacts to sonatype, then someone just has to go click "close and release". I'm happy to do that second part. I think we can make that automated, but I'd be concerned about incorrect artifacts being automatically released. This seems *easier* than the process before to me now that it's setup. 
> 
> Also what's the negative with scpexe? I've released many artifacts with no problem using that? 
> 
> [1] http://www.sonatype.com/people/2009/02/why-putting-repositories-in-your-poms-is-a-bad-idea/
> 
> On Sun, Apr 29, 2012 at 8:48 PM, Steve Swinsburg <steve.swinsburg at gmail.com> wrote:
> (Removed sakai-dev from the convo for the timebeing)
> 
> Do we know what the problem was here? Having to download the entire source goes against what we have been doing with the independent release of projects.
> Was it something to do with the snapshot artifacts being deployed with a uniqueVersion? (ie the timestamp) If so, can't we just set that to false so they aren't suffixed?
> 
> I know that Matt, Noah and others have spent a lot of time getting POMs changed around and setup and am really grateful for all of the work put in, you'll all be getting beers when I see you next,  but I never expected that to happen for 2.9 and now we have a potential issue.
> 
> I'll also put it out there that I think releasing everything to Sonatype is overkill and increases the complexity and steps involved. However if we can get a streamlined process happening then I'll be happier with it.
> 
> When I originally suggested the Sonatype release method (4th Dec 2011, private email to Anthony, Sam and David H), I was talking about the master POM and possibly a condensed purepoms *only*, so that you didn't need a repo in a project POM to be able to find the parent, and more config could be centralised in the parent POMs. 
> 
> I think it is reasonable that we have our own Maven repo for the bulk of the artifacts we create. The issue was the move to Maven3 and the reduced transport protocols that Maven 3 supports (http and file only) and us therefore requiring scpexe extensions to connect to our repo with our SCP credentials.
> 
> If there is an issue with resources for maintaining the infrastructure and expenses for upkeep, I am not aware of that so there may well be good reasons for moving the Maven repo off our own infrastructure.
> 
> cheers,
> Steve
> 
> 
> 
> 
> 
> 
> 
> 
> On 27/04/2012, at 10:45 AM, Matthew Jones wrote:
> 
>> Just as a (short) notice to developers:
>> 
>> A few issues were discussed on this mornings release call. This is mainly for developers working on Sakai CLE trunk, and those who have contrib tools that they are expecting to work on 2.9+.
>> 
>> There were two potential problems which I've resolved tonight that may cause you some issues with your local work.
>> 
>> Quick summary:
>> - If you're using https://source.sakaiproject.org/svn/sakai/trunk/ use https://source.sakaiproject.org/svn/sakai/branches/sakai-trunk-all/ instead.
>> 
>> 1) Sakai 2.10-SNAPSHOT artifacts seemed to still be downloaded even though newer ones were built. It was unsure if this was because of maven version (mvn3 is recommended for trunk development) but as a short term solution, all 2.10 SNAPSHOTS were removed from all repositories and no new artifacts for 2.10 will be deployed.
>> 
>> This means that right now https://source.sakaiproject.org/svn/sakai/trunk/ will probably if you have a clean local repository and will be out of date quickly. 
>> You should instead use https://source.sakaiproject.org/svn/sakai/branches/sakai-trunk-all/. We'll very likely switch trunk with trunk-all.
>> 
>> What this does for you is downloads everything it needs from Sakai rather than downloading assemblies from core-deploy. These assemblies don't entirely make sense in trunk anyway because it changes so often. These assemblies will still be present for Sakai 2.9 and no changes are intended for 2.9 at least until the conference.
>> 
>> Quick summary:
>> - If you have an indie, switch parent from purepoms to master. If you have some apis dependencies inherited through compile scope, these might be overridden by dependencyManagement in master so you may need to define more dependencies.
>> 
>> 2) Purepoms were removed from 2.9 and trunk after 2.9.0-b03 a few months ago. If your contrib tool still uses a purepoms for 2.9 or 2.10 snapshot it will no longer be able to find these artifacts. (https://jira.sakaiproject.org/browse/SAK-21564) Old artifacts for 2.9-SNAPSHOT and 2.10-SNAPSHOT were in source.sakaiproject.org/maven2-snapshots. These artifacts were up to 5 months old. We had changed to deploying all artifacts releases and snapshots to maven central/sonatype, so if your tool had this repository defined you need to update it. 
>> 
>> Ideally your tool can use a fixed release version of 2.9 (like 2.9.0-b05). That would be ideally what your parent looks like: 
>>     <parent>
>>         <groupId>org.sakaiproject</groupId>
>>         <artifactId>master</artifactId>
>>         <version>2.9.0-b05</version>
>>     </parent>
>> You can use the version 2.9-SNAPSHOT which is in the repository, but 2.10-SNAPSHOT currently would require your users to download and build sakai-trunk-all to use. (As mentioned in Step #1)
>> 
>> It was also noticed that some projects had dependencies defined in their "api's" with no scope defined (so it got compile) and when the api was a dependency in something else (like the impl or the tool) those other dependencies would be pulled in as transisitive dependencies. However, since all api's are now defined in the master dependencyManagement as "provided", your tools dependencies may have to have these api's  explicitly defined. 
>> 
>> Confused?
>> 
>> Say you're the dashboard project and you used to inherit from purepoms. Switching to master alone will cause some dependencies to not be found. This is because in the dashboard-api there is:
>> 
>> 	    <dependency>
>> 		    <groupId>org.sakaiproject.assignment</groupId>
>> 		    <artifactId>sakai-assignment-api</artifactId>
>> 	        <version>${sakai.version}</version>
>> 	    </dependency>
>> After switching to master, this scope becomes "provided" instead of "compile" as it isn't defined here, and it won't work when you include it in impl. So you'd need to define these as provided in impl.
>> https://source.sakaiproject.org/viewsvn/longsight/dashboard/trunk/impl/pom.xml?root=contrib&r1=79394&r2=79398
>> _______________________________________________
>> sakai-dev mailing list
>> sakai-dev at collab.sakaiproject.org
>> http://collab.sakaiproject.org/mailman/listinfo/sakai-dev
>> 
>> TO UNSUBSCRIBE: send email to sakai-dev-unsubscribe at collab.sakaiproject.org with a subject of "unsubscribe"
> 
> 
> _______________________________________________
> cle-release-team mailing list
> cle-release-team at collab.sakaiproject.org
> http://collab.sakaiproject.org/mailman/listinfo/cle-release-team
> 
> 

-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://collab.sakaiproject.org/pipermail/cle-release-team/attachments/20120502/5f7e0fc3/attachment-0006.html 