Summary: I would like to know how to build Maven applications consisting of several Maven projects with all inputs from read-only source and all outputs to a given (temporary) folder, without breaking IDE (Netbeans) support. None of the proposed solutions work for me, so I ask in detail to explain my problem as good as possible.
In detail: Since quite a while I try to get my Maven builds integrated in our productive building environment which has several requirements:
- no internet access (all inputs must be under version control)
- source tree is read-only
- all build artifacts must be below a build directory
- well defined toolchain (under version version control or in read-only Virtual Machine)
- libraries shall not know which application use them (i.e. "no parent POM")
- to support development cycle, preferably it should work with IDEs (Netbeans), optional
- to support development cycle, preferably it should work incrementally, optional
This is similar to Out-of-tree build with maven. Is it possible? and other questions related to maven build directories, but unfortunately none of the proposed solutions works here.
I have the "central repository" accessible via "file:" URL and as well as the "local repository" below the (temporary) build directory, but I did not find any way to have mavens "target" directories below the build directory without breaking anything else.
I have several applications that share some libraries. For each library I have an own "parent POM" (violating the DRY principle, because I found no better way). Each contain application and environment specific settings such as the distributionManagement repository path, which is defined using ${env.variable} to allow the build tool to inject the system and application specific values. Default values are provided to make developers using Netbeans happy.
Unfortunately this does not work for build directory.
I could set a build directory as in Maven: How to change path to target directory from command line?. Then all class files (of all libraries and applications) build by one parent POM will be put into one and the same directory. Maven can be ran once and works - but in any later run it will fail! At least unit tests fail, because Maven (surefire) will find all tests for all projects, not only the one Maven is currently processing. So it will try to run unit tests from the application while building a library A (which in my case fails because it also needs library B which is not provided as dependency in library A).
I also tried constructions like "-DbuildDirectory=$BUILDDIR/maven-target/\${project.name}". This works for compilation of sources, but again not for tests. Although Maven (3.1.1) evaluates ${project.name} correctly when storing the files, it passes it literally (!) to the classpath, so when compiling unit test java doesn't find the test objects ("error: cannot find symbol"). Having a symlink ${project.name} to the correct name of course works only for one project (library A for example), because I found no way how to change it from within a pom.xml.
Currently my closest approach is using a global target directory and clean it before each build, but obviously this is bad when developing using this build system (working incrementally without IDE).
Now I'm even considering generating the pom.xml files from the build system with filled values and copy all sources to the build tree. This should work for our defined release builds, but be uncomfortable during normal development (with IDE).
Is there any trick or is this really impossible with Maven?
EDITED to answer the questions
Thanks so much for spending time on my issue / help request! I answer as clearly as possible.
A team mate suggested to add the motivation for all this:
It should be possible to reproduce any release in ten or 20 years, even if internet changed.
Why do you need to change the value of "target"?
Because the source tree is read-only, Maven cannot create the directory at its default position. A writable build directory is available, but outside the source code tree.
Maven will not write to /src, so no problem there
"source tree" does not reference to the "src" folder within the sources / inputs for the build process, but to the whole structure - including POM files, resources and even binary libs if needed. So the directory that contains src also contains pom.xml and belongs to read-only VCS controlled "source tree".
Could you give an example of something that does not work with the standard project layout?
One different example of such a read-only source tree could be: burn a whole project structure (application with libaries) on a CD-R and ensure "mvn compile test package deploy" works.
(One motivation for the read-only source tree requirement is to ensure that the build process does not accidentally (or intentionally) manipulate source code, which could lead to automatically changing build artifacts by the number of repeats, breaking reproducibility.)
using a repository manager
As I understood, a repository manager is a complex service typically running on a host. I think using file URLs instead of such a complex service keeps the dependencies smaller. Please note that the whole repository has to be version controlled, so automatically updating must not work anyway, but as I understand is the advantage of a repository manager.
Do you use dependency jar's
Yes, sure. The libraries and applications depend on several JARs. Even Maven depends on several JARs, such as the compiler plugin.
If your request states that you need to build all dependencies yourself ?
Actually I'm afraid that this could be the case and yes, I'm aware this would be huge effort, unfortunately. Even with Debian packages this was quite difficult to realize and they put a lot of effort to make it possible. At the moment I don't see a realistic chance to build these libraries, unfortunately.
"...all build artifacts must be below a build directory"- Can you please explain that in detail?
The build process is supposed to create all files below a given build directory, for example /tmp/$USER/build/$PROJECTNAME/$PROJECTVERSION/$APPLICATION/$VARIANT/. The build system can create arbitrary structures below that, but is not supposed to change anything outside this. A list of files within this tree is defined as output (build result) and taken from there. Typically some binaries or ZIP files are copied to version control system.
Some call this "out of tree build", "build out of source" or similar.
well defined toolchain
The idea is to have a Virtual Machine image (under version control), for each build create a dynamic "clone" of it, install needed toolchains (from version control), for example the build tools, the Maven installation and its repository, give access to the specific version of the sources to be built (for example, in form of a read-only ClearCase view), run a build script entry point script, collect created outputs (for example some "release.zip" and "buildlogs.zip") and finally discard the whole virtual machine including all its contents. Of course the toolchains could be fully preinstalled in the image, it is just not done for practical reasons (and yes we also use Docker).
For impact analysis it's very important thing to know who is using which library ?
Yes, this surely is very true. However, the sources should abstract from it, and usually do. For example, the authors of JUnit dont' know everybody who is using it. We like to use a library in several projects at the same time, so we cannot mention a parent POM, because there are multiple parent POMs.
I hope I was able to explain better this time.
EDIT #2 to answer the new questions
Thanks again for your time getting through this long post! I hope this time I was able to explain the last missing bits and make it clear. I think some of the questions seem to be comments about the requirements. I'm afraid we could slip into a discussion.
In maven you have a directory structure like:
root +- pom.xml +- srcThe root directory is under version control. So I don't understand the following:
"source tree" does not reference to the "src" folder within the sources / inputs for the build process, but to the whole structure -
root
+- pom.xml
+- src
+-- ...
+- target
+-- ...
I added target" to the structre to better illustrate the problem. As you say, the root directory is under version control and thus read-only. Thus, root/pom.xml, root/src and root itself are read-only.
When Maven would try to create root/target, it will get an error (Read-only file system), because root and all its sub-folders are read-only.
Here even the parent folder is under version control and in case of ClearCase, there are even more additional read-only parent directories, but I think this does not matter here.
We are talking about terra bytes of artifacts....They have to be backed up but not checked into version control.
This is out of scope, but I try to answer nevertheless. The one ClearCase instance I'm working on right now for example has 2.2 TB, so tera bytes, exactly. Backup may not be sufficient if standards require traceability of all changes between releases, for example. However I think this is out of scope of my question.
a file base repository does not work well and needed to be copied to each machine where you use it which is a hassle having a kind of network file system which is mounted to many machines. A repository manager works http(s) based which much more simpler to handle..
I think this is out of scope and not related to my question, but I try to answer anyway. The problem is that for reproducibility, you would need to archive everything of this http(s) for the next ten or 20 years and keep it running. With history, because it might be required to reproduce a five year old state. Of course the needed packages might not be available in the internet anymore (remember Codehaus.org?). Challenging tasks.
a repository manager is much more simpler and keeps the conventions
Unfortunately it does not fulfill the requirements, or require a high price (managing additional virtual machines plus version control of the stored packages).
building all dependencies yourself I think that is simply not worth the effort. I don't a real advantage of that
I think such requirements are not uncommon (https://en.wikipedia.org/wiki/Source_code_escrow), but I'm afraid we are getting off topic and start discussing requirements.
If the JUnit team can access the request from maven central (which can be done) they can see who is using JUnit
I think its off-topic, but how should JUnit developers be able to know all "my" projects that use JUnit?
Where is the relationship to the parent pom?
I'm sorry if I caused confusion, this was a side note only. I think it is out of scope of my question. I'll try to answer nevertheless.
To avoid redundancy, one approach in Maven is using a parent POM and inherit from it. For example, you can define distributionManagement in a parent POM and make the library POMs inherit from it. When you have project specific distributionManagement requirements, you need project specific parent POM files. If two projects share one and the same library, which of the two project POMs should the library inherit (having as parent)? Having the setting distributionManagement in an aggregator POM does not work as it is not propagated.
EDIT #3 to explain read-only
Hi @JF Meier
thank you for your comment.
Read-only means it cannot be written to, so Maven cannot create the target directory. Because of this, javac cannot store the class files and compilation aborts.
Maybe it is too abstract, so let me give a full real example:
cat pom.xml:
<?xml version="1.0" encoding="UTF-8"?>
<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
<modelVersion>4.0.0</modelVersion>
<groupId>test</groupId>
<artifactId>mavenproject1</artifactId>
<version>1.0-SNAPSHOT</version>
</project>
cat src/main/java/Hello.java:
public class HelloWorld {
public static void main(String[] args) { System.out.println("Hello, World!"); }
}
Now copy & paste from the shell (I cut a few boring lines):
steffen@node1:/view/dev_test_project_v1/vobs/playground/steffen/minimal_pom $ mvn compile
[...]
[ERROR] Failure executing javac, but could not parse the error:
javac: directory not found: /view/dev_test_project_v1/vobs/playground/steffen/minimal_pom/target/classes
This is one problem. A small detail I have to correct, it is not wshowing the Read-only file sytem error, I think a bug, but in strace we can see it:
21192 mkdir("/view/dev_test_project_v1/vobs/playground/steffen/minimal_pom/target/classes", 0777) = -1 ENOENT (No such file or directory)
21192 mkdir("/view/dev_test_project_v1/vobs/playground/steffen/minimal_pom/target", 0777) = -1 EROFS (Read-only file system)
21192 mkdir("/view/dev_test_project_v1/vobs/playground/steffen/minimal_pom/target", 0777) = -1 EROFS (Read-only file system)
This leads to the javac: directory not found error.
It is read-only:
steffen@node1:/view/dev_test_project_v1/vobs/playground/steffen/minimal_pom $ mkdir -p target/classes
mkdir: cannot create directory ‘target’: Read-only file system
The mkdir tool correctly shows the error message.
