Build Gradle repository for offline development

Question

I am working on implementing a Gradle build system for a piece of software that has parts that are developed in area without Internet connectivity or the ability to install a Maven/Ivy server (like Nexus). To support development in these environments, I am putting together a Gradle plugin that allows the generation of an "Offline Workspace".

I originally implemented this functionality by triggering the resolution of each configuration in the project (triggering the download of all dependencies), then traversing the entire dependency tree of each configuration and copying the local cached copy of the dependency into the Offline Workspace. (A Copy task was generated for each copy operation.) These JARs would then be referenced using a flatDir repository.

This implementation performed its job using an afterEvaluate block. While this worked fine in Gradle 2.0, it triggers a deprecation warning in Gradle 2.2.1 because triggering the resolution is somehow seen as modifying a configuration after it has already been resolved (Attempting to change configuration ':core:runtime' after it has been included in dependency resolution. This behaviour has been deprecated and is scheduled to be removed in Gradle 3.0). In all, this approach feels rather hacky since it also requires me to modify the build.gradle files to explicitly list all transitive dependencies since there are no POM files available to properly specify dependencies.

A more elegant approach seems like it would build a local Maven repository of all dependencies (including POM files, source JARs, javadoc JARs, etc) and then just use the mavenLocal() repository type. Unfortunately, I'm not sure how to do this properly where I don't need to trigger artifact resolution in order to perform this operation.

Is there some better way that I can achieve the full artifact download into an easy-to-package way than just zipping up my entire $USER_HOME/.gradle directory?

Answer 1

To have an offline build you need somehow to provide all required dependencies. One of the options here is just to commit those jars into version control. The hard part is to collect all those dependencies. For that it's possible to have a build.gradle file that can operate in two modes (online and offline):

buildscript {
    repositories {
        if ('allow' == System.properties['build.network_access']) {
            mavenCentral()
        } else {
            maven {
                url 'dependencies'
            }
        }
    }
    dependencies {
        classpath 'com.android.tools.build:gradle:1.2.0-beta2'
    }
}

To run in offline mode type:

gradle --offline

And to run in online mode:

gradle -Dbuild.network_access=allow

And to collect all dependencies use this script that will run gradle in online mode, fetch dependencies to cache inside ${project_dir}/.gradle_home and copy artifacts to local maven repository in dependencies folder.

#!/usr/bin/python

import sys
import os
import subprocess
import glob
import shutil

# Place this in build.gradle:
# repositories {
#     if ('allow' == System.properties['build.network_access']) {
#         mavenCentral()
#     } else {
#         maven { url 'dependencies' }
#     }
# }
def main(argv):
    project_dir = os.path.dirname(os.path.realpath(__file__))
    repo_dir = os.path.join(project_dir, "dependencies")
    temp_home = os.path.join(project_dir, ".gradle_home")
    if not os.path.isdir(temp_home):
        os.makedirs(temp_home)
    subprocess.call(["gradle", "-g", temp_home, "-Dbuild.network_access=allow"])
    cache_files = os.path.join(temp_home, "caches/modules-*/files-*")
    for cache_dir in glob.glob(cache_files):
        for cache_group_id in os.listdir(cache_dir):
            cache_group_dir = os.path.join(cache_dir, cache_group_id)
            repo_group_dir = os.path.join(repo_dir, cache_group_id.replace('.', '/'))
            for cache_artifact_id in os.listdir(cache_group_dir):
                cache_artifact_dir = os.path.join(cache_group_dir, cache_artifact_id)
                repo_artifact_dir = os.path.join(repo_group_dir, cache_artifact_id)
                for cache_version_id in os.listdir(cache_artifact_dir):
                    cache_version_dir = os.path.join(cache_artifact_dir, cache_version_id)
                    repo_version_dir = os.path.join(repo_artifact_dir, cache_version_id)
                    if not os.path.isdir(repo_version_dir):
                        os.makedirs(repo_version_dir)
                    cache_items = os.path.join(cache_version_dir, "*/*")
                    for cache_item in glob.glob(cache_items):
                        cache_item_name = os.path.basename(cache_item)
                        repo_item_path = os.path.join(repo_version_dir, cache_item_name)
                        print "%s:%s:%s (%s)" % (cache_group_id, cache_artifact_id, cache_version_id, cache_item_name)
                        shutil.copyfile(cache_item, repo_item_path)
    shutil.rmtree(temp_home)
    return 0

if __name__ == "__main__":
    sys.exit(main(sys.argv))

So, after each dependency change just run this script and commit changes in dependencies folder. Then you can build offline with gradle --offline or just gradle.

Answer 2

In the past, I used a similar solution, though "my" copy script was done in groovy instead of python.

I detected another approach a couple of weeks ago: There is the ivypot plugin. Now you don't have to use any "external" script any more, the plugin is able to copy all dependencies into a local directory which is an ivy repo.

A test project can be found on github. I can provide an English README in case there is demand for it.

Answer 3

The most straight-forward solution is to snapshot the entire dependency cache directory: ~/.gradle

There are some challenges though:

1. It will take a significant amount of disk space
2. It will contain thousands of files (difficult to manipulate manually)
3. Since this is a cache not a repository, Gradle is under no obligation to keep things permanently

To address items 2 and 3 above, I suggest using Git (and also backup copies) to help you defeat Gradle's cache-cleaning daemon and avoid the error: "no cached version available for offline mode". When you have a fully-populated dependency cache (--offline builds are working), commit the cache to version control so that you can restore it later if necessary (using git stash to discard all changes, for example).

---

The ~/.gradle folder should still work as a symbolic link to elsewhere in your filesystem if that simplifies project management or backups. There may also be some improvement of this scheme available by using a --project-cache-dir, but I haven't yet attempted this.

---

This involves some manual repository management, since Gradle cache management is actively working against the goal of stable offline development -- but it should help keep you running.