Optional testFixtures in Gradle

Question

I would like to add the java-test-fixtures plugin to a custom conventions plugin under buildSrc that I simply add to any of my modules. By doing so I only need to add my custom conventions plugin instead of having multiple (and different) plugins for each sub-module. By doing so the build won't work anymore if a sub-module does not have a test fixtures source set (/src/testFixtures/java/...). Is there any way in Gradle to make the usage of java-test-fixtures optional?