Where should the integration tests be stored when using the maven-failsafe-plugin?

Question

Do I have to place my integration tests under src/test with the rest of my unit tests and just distinguish them by a pattern such as *Integr*Test, *ITTest, or can they be in src/it (as is the case when developing Maven plugins and using the maven-invoker-plugin)?

I'm asking this because, to me it looks not clean enough if both unit and integration tests are in the same place, (even if they were to be controlled via a Maven profile).

What you mentioned it is not clean is correct cause those IT get the classpath as your unit tests which might not what you like. So best is to have complete separate module which contains the IT's. Locate them into using `src/test/java` and follow the naming convention for IT's as you already mentioned. — khmarbaise, Jun 24 '16 at 06:20
@khmarbaise: Thanks for the explanation, I appreciate it's coming straight from one of the commiters on the Apache Maven project! I understand what you're saying, although I don't quite agree that if you had it in a different directory, your `test` dependencies will be different. They won't be. What I also don't understand, (despite having read quite a bit on the subject), is -- why there needs to be a separate plugin for this in this case? There could simply have been another goal for the `maven-surefire-plugin` that would get invoked during the integration testing phase? — carlspring, Jun 24 '16 at 09:50

score 16 · Answer 1 · answered Jun 23 '16 at 19:14

You are right that src/it is intended to be used for integrations test of plugins. This is mentioned in the Standard Directory Layout.

The maven-failsafe-plugin, by default, will look for your integration tests inside ${project.build.testSourceDirectory}, which is the same as the maven-surefire-plugin for unit tests. By default, this corresponds to src/test/java. The integration tests are made distinct by following a naming convention:

<includes>
  <include>**/IT*.java</include>
  <include>**/*IT.java</include>
  <include>**/*ITCase.java</include>
</includes>

which is different than the naming convention for unit-tests:

<includes>
  <include>**/Test*.java</include>
  <include>**/*Test.java</include>
  <include>**/*TestCase.java</include>
</includes>

So while they would reside in the same source folder (src/test/java), the difference in names clearly distinguishes them. Also, this is the default set-up so no extra configuration would be needed.

That said, you can have other options:

Place the integration tests inside a different source folder. This will require some configuration to make it work: you will need to use the build-helper-maven-plugin:add-test-source goal to add the custom folder as a test source folder.
Use a different module (if you have a multi-module Maven project) that would only contain the integration tests.

Thanks for your detailed explanation! It clears out a lot! I know it should probably be in another question, but could you maybe also reply to my question in the comments above to @khmarbaise? — carlspring, Jun 24 '16 at 09:52
@carlspring I'm afraid I don't know the exact reason for creating a new plugin for ITs. My best guess would be because ITs are simply very different than unit tests: integration tests have a pre and post phases, whereas unit tests don't; typically ITs require to set-up an custom environment to run them (launching a server for example). I personally wouldn't have ITs in a different directory: it requires some config. IMO, having a separate module is a clean solution and is perfect when you already have a multi-module project (I'm not sure I'd go for the module approach otherwise). — Tunaki, Jun 24 '16 at 10:06
Thanks! This makes sense as an explanation, although I'm sure a couple of extra goals to the `maven-surefire-plugin` would have had pretty much the same effect. But, perhaps, there are implications I am not aware of yet. Once again, thanks for your time! — carlspring, Jun 24 '16 at 10:11

score 4 · Accepted Answer · edited Aug 21 '23 at 13:53

First maven-fails-plugin runs by default in another life cycle phase (integration-test) as maven-surefire-plugin (test) does. Furthermore you can configura maven-failsafe-plugin to run verify goal at the post-integration-test test phase if you like to check if integration tests have failed. This can be freely configured.

There is one question coming into my mind. You have 10 modules and now you would like to have integration tests? To which module do they belong? So best is to have a separate module cause they belong to none of the 10 modules.

Apart from that maven-surefire-plugin is already configured in default life cycle. Yes a supplemental goal would be an idea, but it would confuse user to use the same plugins in different relationships. So separation of concerns is important here. Apart from the whole default configurations...Those plugins share a larger code base but there are differences...

Also what has already been mentioned by Tunaki is pre-integration-test, for setup things like servers etc. integration-test and things like shutting down services/servers in post-integration-test phase. This will never happen in unit tests.

Using a separate module makes it usually simpler to setup the IT's which means having different dependencies (classpath) than for the unit tests. For example things like Arquillian.org which is never using in unit tests. This can't be handled in a single module...also a good things is separation of concerns here..

Furthermore integration tests could not be parallized by default whereas unit tests can be by definition otherwise they not unit tests.

So what about folder layout? In the integration test module you can simply use the src/test/java folder which means you don't need supplemental configuration etc. (for example via build-helper-maven-plugin etc.) which makes it easier and follows more the convention over configuration paradigm.

And not to forget you can better control what is running in your build (CI)..

And another important hint. Usually integration tests are often related to infrastructure so it might sometimes useful to ignore failures there which can simply handle by using check goal of maven-failsafe-plugin....

An example for an IT module can be found here.

Where should the integration tests be stored when using the maven-failsafe-plugin?

2 Answers2

Linked