Welcome to the fun world of sharing outputs between Gradle subprojects!
There's a few different ways of solving this problem.
Since you're working with a JVM project you might want to configure a feature variant. Gradle will then create a new source set, and you can generate your files into this.
But, if you're not using a JVM project, then things get a little bit more complicated. I'll explain the simplest and most straight forward way. Please stop laughing, that wasn't a joke. Yes I know, it's still complicated. Trust me, it's worth it.
What we'll end up with is a robust way to share files between projects in a way that's cacheable (being compatible with both Build Cache and Configuration Cache), flexible, re-usable, and gain a good understanding of how Gradle works.
Tasks
Here's the summary of what needs to be done:
- Create a buildSrc convention plugin
- Set up some configurations* for providing and consuming files
- Define a custom variant attribute as a marker, to differentiate our configurations from others
- Put the files produced by a task into the outgoing configuration
- Resolve the files from other subprojects using the incoming configuration
*Yes, the name configuration is confusing. In this context 'configuration' should be better renamed as 'DependencyContainer' - they're just a collection of files that might be outgoing or incoming to a subproject, along with some metadata to describe the contents.
Creating a buildSrc convention plugin
We need to be able to set-up our providing and consuming code in both the providing and consuming subprojects. While technically we could copy-paste it, that kind sucks and far too finickity. The best way to share config in Gradle is with a convention plugin.
I've gone over setting up convention plugins in another answer (Configure Kotlin extension for Gradle subprojects), so I'll just summarise the steps here.
Create the build config for buildSrc.
Since buildSrc is effectively a standalone project, it's best to create a settings.gradle.kts file. I like using a centralised repositories declaration.
// buildSrc/settings.gradle.kts
rootProject.name = "buildSrc"
pluginManagement {
repositories {
mavenCentral()
gradlePluginPortal()
}
}
@Suppress("UnstableApiUsage")
dependencyResolutionManagement {
repositoriesMode.set(RepositoriesMode.PREFER_SETTINGS)
repositories {
mavenCentral()
gradlePluginPortal()
}
}
The build.gradle.kts just needs the Kotlin DSL plugin. This will make writing our convention plugin much more easy as precompiled script plugin
// buildSrc/build.gradle.kts
plugins {
`kotlin-dsl`
}
Create our convention plugin. The name of the file (everything before .gradle.kts), and the package (if one is defined) will be the plugin ID
// buildSrc/src/main/kotlin/generated-files-sharing.gradle.kts
logger.lifecycle("I don't do anything yet...")
Done! You can test it by applying this do-nothing plugin to your subprojects.
// my-generator/build.gradle.kts
plugins {
id("generated-files-sharing")
}
// my-consumer/build.gradle.kts
plugins {
id("generated-files-sharing")
}
When you run a task (e.g. ./gradlew help), you should see the message I don't do anything yet... logged to the console.
Creating configurations for providing and consuming files
The next step is creating some configurations, which are the shipping containers of Gradle's dependencies world.
We're going to make two configurations, one for incoming files, and another for outgoing files.
Take note of isCanBeConsumed and isCanBeResolved*. In Gradle terminology, the incoming one will be resolved by a subproject, and the outgoing one will be consumed by other subprojects. It's important to use the right combination.
*Again, we have some confusing names. The terms 'consumed' and 'resolved' aren't very clear, they both seem like synonyms to me. They would be better renamed to indicate that consumed=true && resolved=false means OUTGOING, and consumed=false && resolved=true means INCOMING
// buildSrc/src/main/kotlin/generated-files-sharing.gradle.kts
// register the incoming configuration
val generatedFiles by configurations.registering {
description = "consumes generated files from other subprojects"
// the dependencies in this configuration will be resolved by this subproject...
isCanBeResolved = true
// and so they won't be consumed by other subprojects
isCanBeConsumed = false
}
// register the outgoing configuration
val generatedFilesProvider by configurations.registering {
description = "provides generated files to other subprojects"
// the dependencies in this configuration won't be resolved by this subproject...
isCanBeResolved = false
// but they will be consumed by other subprojects
isCanBeConsumed = true
}
What's cool about this step is that if you now jump to your subproject you can use these configurations as if they were built into Gradle (after an IDE sync, if necessary).
// my-consumer/build.gradle.kts
plugins {
id("generated-files-sharing")
}
dependencies {
generatedFiles(project(":my-generator"))
}
Gradle has generated a type-safe accessor for the generatedFiles configuration.
However, we're not done. We haven't put any files into the outgoing configuration, so of course the consuming project isn't going to be able to resolve any files. But before we do that, we need to add that metadata I mentioned.
Differentiate our configurations with variant attributes
It's quite likely that the subproject that provides the generated files also has other configurations with completely different file types. But we don't want to fill up the generatedFiles configuration with lots of files, we only want generated files produced by our generate task.
That's where variant attributes come in. If configurations were shipping containers, then the variant attributes are the shipping labels on the outside that describe where the contents should be sent.
At a basic level, variant attributes are just key-value strings, except the keys have to be registered with Gradle. We can skip that registration though if we use a built-in standard attribute that we can use. The Usage attribute is a good pick. It's pretty commonly used, so long as we pick a distinctive value Gradle is going to be able to differentiate between two configurations by comparing the values.
// buildSrc/src/main/kotlin/generated-files-sharing.gradle.kts
// create a custom Usage attribute value, with a distinctive value
val generatedFilesUsageAttribute: Usage =
objects.named<Usage>("my.library.generated-files")
val generatedFiles by configurations.registering {
description = "consumes generated files from other subprojects"
isCanBeResolved = true
isCanBeConsumed = false
// add the attribute to the incoming configuration
attributes {
attribute(Usage.USAGE_ATTRIBUTE, generatedFilesUsageAttribute)
}
}
val generatedFilesProvider by configurations.registering {
description = "provides generated files to other subprojects"
isCanBeResolved = false
isCanBeConsumed = true
// also add the attribute to the outgoing configuration
attributes {
attribute(Usage.USAGE_ATTRIBUTE, generatedFilesUsageAttribute)
}
}
What's important is that the same attribute key and value are added to both configurations. Now, Gradle can happily match both the incoming and outgoing configurations!
Now all the parts are in place. We're almost done! It's time to start pushing files into the configuration, and pulling files out.
Putting files into the outgoing configuration
In the subproject that produces the generated files, let's say we have some task that produces some files. I'll use a Sync task as a stand-in for the actual generator task.
// my-generator/build.gradle.kts
plugins {
id("generated-files-sharing")
}
val myGeneratorTask by tasks.registering(Sync::class) {
from(resources.text.fromString("hello, world!"))
into(temporaryDir)
}
Note that the output directory doesn't really matter, because thanks to Gradle's Provider API, a task can be converted into a file-provider.
// my-generator/build.gradle.kts
configurations.generatedFilesProvider.configure {
outgoing {
artifact(myGeneratorTask.map { it.temporaryDir })
}
}
What's nice about this is that now Gradle will only configure and run the myGeneratorTask task if it requested. When this kind of configuration avoidance is regularly used, then it can really help speed up Gradle builds.
Resolving the incoming configuration
We're at the last step!
In the consuming project we can add a dependency on the providing project using the regular dependencies {} block.
// my-consumer/build.gradle.kts
plugins {
id("generated-files-sharing")
}
dependencies {
generatedFiles(project(":my-generator"))
}
And now we can get the incoming files from the incoming configuration:
// my-consumer/build.gradle.kts
val myConsumerTask by tasks.registering(Sync::class) {
from(configurations.generatedFiles.map { it.incoming.artifacts.artifactFiles })
into(temporaryDir)
}
Now if you run ./gradlew myConsumerTask you'll notice that, even though you haven't explicitly set any task dependencies, Gradle will automatically run myGeneratorTask.
And if you check the contents of myConsumerTask's temporary directory (./my-consumer/build/tmp/myConsumerTask/), you'll see the generated file.
If you re-run the same command, then you should see that Gradle will avoid running the tasks because they are UP-TO-DATE.
You can also enable Gradle Build Cache (./gradlew myConsumerTask --build-cache) and even if you delete generated files (remove the ./my-generator/build/ directory) you should see that both myGeneratorTask and myConsumerTask are loaded FROM-CACHE
