[cle-release-team] [Building Sakai] Important information for trunk developers (SNAPSHOT artifacts)

[Matthew Jones]

I have a feeling that the real problem was that there were a few projects
(search, basiclti) where the poms were messed up, and it was only
(obviously) uncovered after removing all of the snapshot artifacts. Aaron
has been commenting about it for awhile but I was never able to reproduce
his problem for whatever reason. As soon as all of the artifacts were
removed, I was able to find and fix a number of issues with the build.

Search was a problem because the assembly had dependencies on test
artifacts which were not built when skip.test=true (because of a profile in
the root pom for search). I think this more showed which developers were
skipping tests verses which developers actually had problems though. ;)

BasicLTI had a problem because the release plugin bumped the version (to
2.1-SNAPSHOT), which wasn't correctly updated in master. Since the (old)
snapshot artifacts (for 2.0-SNAPSHOT) were in the repository, it was still
using this version for something that depended on it. So everyone that was
pulling BasicLTI in a regular trunk build was pulling the wrong version for
the last 5 months.

I agree requiring a "-all" build goes against the point of indies, but
there was some disagreement about if every tool that currently *is* indie
really should be an indie, if it makes sense both as far as the amount of
time it takes one person to release, manage and track everything everything
everything. This is balanced for all of the institutions that want to
feature patch indies. They need to either checkout the "-all" or do some
process that we don't have documented. We don't have a document (AFIAK)
that says:

- If you as an institution want to make local patches (pull in features
specifically) against a tool that is an indie (downloaded by core-deploy)
what should you do?
- How about if you want to pull in local patches against a library? (Like
jsf? edu-services? kernel?)
- How are you sure your changes *are* in there for sure and maven didn't
download artifacts/assemblies? (Other than parsing the build logs all the
time or running in --offline mode?)

These were questions I wasn't ready to answer last Thursday, at least for
trunk. These were all issues I wanted to either put off till the conference
or at least try removing all artifacts to "see what happened". Generally it
seems like if nothing "seems" to be broken than there's no discussion. As
soon as it's broke, then people come out of nowhere. ;)

All 2.9 indies SNAPSHOTS and release *are* in sonatype. The decision to
remove purepoms retroactively in 2.9 rather than waiting until 2.10 seemed
like a good one. Purepoms only served to complicate the build order,
complicated having to redefine and keep properties up to date and only a
handful of people knew which purepom they were actually supposed to use.
Though Seth suggested keeping them at least around as SNAPSHOTS (which I
did), but it doesn't make any sense to release them, since nobody will
reference released versions.

It's much easier just using master, it's essentially the same, it just has
more properties and dependencyManagement defined. If you don't like the
properties and dependencyManagement in master, just override it locally.
But it's should be a "good default".

I'm not sure if there was a significant concern about the lack of server
resources, but just strong recommendations from sonatype docs that all
artifacts should just be in maven central (since this is in the master
maven pom forever) [1] Since we were already deploying some things there,
it seemed easier to just deploy everything there. This was a decision made
on the release call. back in February.

Which part of it do you want more streamlined? Jenkins, when you click
"Perform Release" and fill in the info, release the artifacts to sonatype,
then someone just has to go click "close and release". I'm happy to do that
second part. I think we can make that automated, but I'd be concerned about
incorrect artifacts being automatically released. This seems *easier* than
the process before to me now that it's setup.

Also what's the negative with scpexe? I've released many artifacts with no
problem using that?

[1]
http://www.sonatype.com/people/2009/02/why-putting-repositories-in-your-poms-is-a-bad-idea/

On Sun, Apr 29, 2012 at 8:48 PM, Steve Swinsburg
<steve.swinsburg at gmail.com>wrote:

> (Removed sakai-dev from the convo for the timebeing)
>
> Do we know what the problem was here? Having to download the entire source
> goes against what we have been doing with the independent release of
> projects.
> Was it something to do with the snapshot artifacts being deployed with a
> uniqueVersion? (ie the timestamp) If so, can't we just set that to false so
> they aren't suffixed?
>
> I know that Matt, Noah and others have spent a lot of time getting POMs
> changed around and setup and am really grateful for all of the work put in,
> you'll all be getting beers when I see you next,  but I never expected that
> to happen for 2.9 and now we have a potential issue.
>
> I'll also put it out there that I think releasing everything to Sonatype
> is overkill and increases the complexity and steps involved. However if we
> can get a streamlined process happening then I'll be happier with it.
>
> When I originally suggested the Sonatype release method (4th Dec 2011,
> private email to Anthony, Sam and David H), I was talking about the master
> POM and possibly a condensed purepoms *only*, so that you didn't need a
> repo in a project POM to be able to find the parent, and more config could
> be centralised in the parent POMs.
>
> I think it is reasonable that we have our own Maven repo for the bulk of
> the artifacts we create. The issue was the move to Maven3 and the reduced
> transport protocols that Maven 3 supports (http and file only) and us
> therefore requiring scpexe extensions to connect to our repo with our SCP
> credentials.
>
> If there is an issue with resources for maintaining the infrastructure and
> expenses for upkeep, I am not aware of that so there may well be good
> reasons for moving the Maven repo off our own infrastructure.
>
> cheers,
> Steve
>
>
>
>
>
>
>
>
> On 27/04/2012, at 10:45 AM, Matthew Jones wrote:
>
> Just as a (short) notice to developers:
>
> A few issues were discussed on this mornings release call. This is mainly
> for developers working on Sakai CLE trunk, and those who have contrib tools
> that they are expecting to work on 2.9+.
>
> There were two potential problems which I've resolved tonight that may
> cause you some issues with your local work.
>
> *Quick summary:*
> - If you're using https://source.sakaiproject.org/svn/sakai/trunk/ use
> https://source.sakaiproject.org/svn/sakai/branches/sakai-trunk-all/
>  instead.
>
> 1) Sakai 2.10-SNAPSHOT artifacts seemed to still be downloaded even though
> newer ones were built. It was unsure if this was because of maven version
> (mvn3 is recommended for trunk development) but as a short term solution,
> all 2.10 SNAPSHOTS were removed from all repositories and no new artifacts
> for 2.10 will be deployed.
>
> This means that right now https://source.sakaiproject.org/svn/sakai/trunk/ will
> probably if you have a clean local repository and will be out of date
> quickly.
> You should instead use
> https://source.sakaiproject.org/svn/sakai/branches/sakai-trunk-all/.
> We'll very likely switch trunk with trunk-all.
>
> What this does for you is downloads everything it needs from Sakai rather
> than downloading assemblies from core-deploy. These assemblies don't
> entirely make sense in trunk anyway because it changes so often. These
> assemblies will still be present for Sakai 2.9 and no changes are intended
> for 2.9 at least until the conference.
>
> *Quick summary:*
> - If you have an indie, switch parent from purepoms to master. If you have
> some apis dependencies inherited through compile scope, these might be
> overridden by dependencyManagement in master so you may need to define more
> dependencies.
>
> 2) Purepoms were removed from 2.9 and trunk after 2.9.0-b03 a few months
> ago. If your contrib tool still uses a purepoms for 2.9 or 2.10 snapshot it
> will no longer be able to find these artifacts. (
> https://jira.sakaiproject.org/browse/SAK-21564) Old artifacts for
> 2.9-SNAPSHOT and 2.10-SNAPSHOT were in
> source.sakaiproject.org/maven2-snapshots. These artifacts were up to 5
> months old. We had changed to deploying all artifacts releases and
> snapshots to maven central/sonatype, so if your tool had this repository
> defined you need to update it.
>
> Ideally your tool can use a fixed release version of 2.9 (like 2.9.0-b05).
> That would be ideally what your parent looks like:
>
>     <parent>
>         <groupId>org.sakaiproject</groupId>
>         <artifactId>master</artifactId>
>         <version>2.9.0-b05</version>
>     </parent>
>
> You can use the version 2.9-SNAPSHOT which is in the repository, but
> 2.10-SNAPSHOT currently would require your users to download and build
> sakai-trunk-all to use. (As mentioned in Step #1)
>
> It was also noticed that some projects had dependencies defined in their
> "api's" with no scope defined (so it got compile) and when the api was a
> dependency in something else (like the impl or the tool) those other
> dependencies would be pulled in as transisitive dependencies. However,
> since all api's are now defined in the master dependencyManagement as
> "provided", your tools dependencies may have to have these api's
> explicitly defined.
>
> Confused?
>
> Say you're the dashboard project and you used to inherit from purepoms.
> Switching to master alone will cause some dependencies to not be found.
> This is because in the dashboard-api there is:
>
>
> 	    <dependency>
> 		    <groupId>org.sakaiproject.assignment</groupId>
> 		    <artifactId>sakai-assignment-api</artifactId>
> 	        <version>${sakai.version}</version>
> 	    </dependency>
>
> After switching to master, this scope becomes "provided" instead of
> "compile" as it isn't defined here, and it won't work when you include it
> in impl. So you'd need to define these as provided in impl.
>
> https://source.sakaiproject.org/viewsvn/longsight/dashboard/trunk/impl/pom.xml?root=contrib&r1=79394&r2=79398
> _______________________________________________
> sakai-dev mailing list
> sakai-dev at collab.sakaiproject.org
> http://collab.sakaiproject.org/mailman/listinfo/sakai-dev
>
> TO UNSUBSCRIBE: send email to
> sakai-dev-unsubscribe at collab.sakaiproject.org with a subject of
> "unsubscribe"
>
>
>
> _______________________________________________
> cle-release-team mailing list
> cle-release-team at collab.sakaiproject.org
> http://collab.sakaiproject.org/mailman/listinfo/cle-release-team
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://collab.sakaiproject.org/pipermail/cle-release-team/attachments/20120430/e18a416c/attachment-0006.html