1

Our organization currently uses a sort-of hacked together form of Java dependency management, effectively relying on IDE configurations and then maintaining a repository of these configs that can be checked out. We have a hybrid of dependencies; some of them are .jars from third parties, whereas others are projects (in Eclipse parlance) or modules (in IntelliJ parlance) that are developed internally. We don't have a "customer" that we ship to nor do we have an Open Source community (we're a research lab) so we typically don't generate binaries of any of our work. Basically, all of our internal code is always working from source. This poses some problems, though:

  1. This doesn't translate to our CI server, which means we also have to maintain a repository of Ant scripts identifying source dependencies on the classpath so that Bamboo can figure out build properly. This is annoying.
  2. This method isn't cross-IDE; some of our devs prefer Eclipse while others prefer IntelliJ IDEA. If it'll make our developers more productive by letting them pick their environment of choice, we don't want to get in the way of that.
  3. This method in an of itself is pretty brittle; if somebody makes a new project, then we have to spend a bunch of time fiddling with Bamboo's Ant scripts and re-exporting our IDE configs.

I'd like to move our researchers towards a more Software-Engineer-y approach like automated build/dependency management that would allow for portability between IDE's as well as our CI setup, but it seems from my reading of the tutorials for Maven and Ivy (haven't spent much time looking at other tools yet) that the general idea is to define binary dependencies, with the option to pull down the sources in to your .m2 or .ivy2 as an extra, but not as the sole method of dependency management. Ideally, we would be able to use one of these tools as a way to handle both binary dependencies from Third Parties as well as our internal source dependencies by pointing to our internal version control. Is this possible with the existing tools or is there a tool that I'm not aware of that would allow for this?

Edit

Maybe a better way to state what I'm looking for is not just dependency management for the purpose of triggering a build, but also managing dependencies when initializing a fresh development environment. Check out project A, and your dependency manager checks out all of the relevant local sources and third party .jars; in Java, this isn't that far removed from building since most IDE's are effectively nice GUI's around classpath/dependency management with a fancy text editor thrown in so I didn't think the request was that farfetched, but I may have failed to think about it hard enough.

Community
  • 1
  • 1
Doug Stephen
  • 7,181
  • 1
  • 38
  • 46
  • Please explain a bit what your rationale for trying to include "source dependencies" is. In Java, all of your dependencies have to get compiled down before use anyway; there's no advantage in a "source-only" dependency. It sounds like what you're wanting to do is use a system like [Nexus](http://www.sonatype.org/nexus) with proxying. – chrylis -cautiouslyoptimistic- Sep 04 '13 at 14:37
  • @chrylis Because the source dependencies are internal projects that aren't frozen. It's the equivalent of saying that a project depends on another project in Eclipse, or that a module depends on another module in IntelliJ IDEA. – Doug Stephen Sep 04 '13 at 14:53
  • That's exactly what `snapshot` dependencies in Maven are for. Presumably you still only want to pull new versions that have passed their unit tests, so you just set up your CI tool to watch for new source versions, compile and test, and then push them to Nexus as snapshot updates if they pass. – chrylis -cautiouslyoptimistic- Sep 04 '13 at 15:29
  • @chrylis this is an approach I was considering (having binary snapshots of passing builds), but it would involve reworking our CI builds. I'm not opposed to it but I expect a lot of resistance from the PI's to this approach. I haven't given up on it yet though. Thanks for the tip about Nexus! – Doug Stephen Sep 04 '13 at 15:42
  • The only other possible approach I can think of for what you're describing would be to require the users to pull and then build every new version from source control themselves, which seems like both a massive headache to set up and a recipe for disaster when someone pushes a bad checkin. – chrylis -cautiouslyoptimistic- Sep 04 '13 at 15:44

2 Answers2

1

I'm preeeeetty sure this answer will be incomplete...

It is true maven and ivy and everyone (gradle, sbt, rake) base themselves on binary dependencies - as one will usually run applications based on jar files not on sources.

In addition you can generate source jars to allow others to look inside your work - which is as you say more important for public projects and not so much for only internal ones. So I would recommend to always generate source jars. So they will be archived in a maven repository.

The good news: it doesn't really matter. I think the confusion comes from the responsibilities between the used IDE and the build tool(s). To ease Continuous Integration you need to take the IDE out of the calculation and base the build solely on the build tool. The IDE will then extract the information it needs from the build tool configs and configure your project. Thats why support for your build system is so important in the IDE.

I think what may be an issue here is that you have several projects in several source repositories (SVN, Git, ...) that come together in the IDE. I think thats not too good (if its true). Maven does maintain information on where a project comes from. So in theory you could checkout one, and then retrieve its dependencies and check out those projects too. But that sounds like a Eclipse / Maven Plugin.

In the IDE the integration will determine if you have a project open that matches a dependency (based on the groupdId, artifactId, version - the GAV coordinates) and resolve the sources from there and not loading them from a remote repository.

I think the only bigger change would be how the initial projects are imported into the IDE? And maybe you are able to exctract modules that are not changed every day and treat them as standard dependencies. This speeds up the build as well as it improves the stability of a build.

Hmmmm. Hope that did help in some kind :)

wemu
  • 7,952
  • 4
  • 30
  • 59
  • +1 It's also possible to warp ivy to download source packages and compile them on the fly. Possible, not necessarily recommended :-) See http://stackoverflow.com/questions/9294996/ivy-custom-resolvers-for-git-or-tfs/9351126#9351126 – Mark O'Connor Sep 04 '13 at 22:06
0

I don't think maven prefers binaries over jars of sources. What is that perception based upon? As far as I recall, when you build a jar of sources, via the maven source plugin, it will create an "attached artifact". That artifact is deployed to Nexus ( or whatever repo you are using ) when the build runs thru the deploy phase. Typically, it will sit right next to the jar of class files. They are both maven artifacts and can be referenced independently from one another, as dependencies, in other builds. The key to discriminating between the two is the fourth maven coordinate, the "classifier".

Unless I missing something, this would serve your purposes . . .

chad
  • 7,369
  • 6
  • 37
  • 56
  • I'm using binary as an equivalent to "jar of sources". My point is that I need a directory hierarchy with .java and .class files, not a .jar file. – Doug Stephen Sep 04 '13 at 19:47