5

I have a an android gradle build that looks for dependencies on four different servers.

Now the build takes up to two minutes because those dependencies are checked every build. Maybe this occurs because one of the repositories returns an error for most of the dependencies.

I have the following sources in my gradle file: 

repositories {
   maven { url 'http://jenkins.local:8081/nexus/content/groups/public/' }
   mavenCentral()
   maven { url 'https://oss.sonatype.org/content/repositories/snapshots/'}
   maven {url "https://repo.commonsware.com.s3.amazonaws.com"}
}

The first entry is a proxy that should cache all artifacts after the first request. This is a part of the dependency list:

compile 'com.android.support:support-v4:19.+'

compile 'com.squareup.picasso:picasso:2.2.+'
compile 'com.commonsware.cwac:merge:1.0.1'
compile files('libs/crittercism_v4_3_0_sdkonly.jar')
testCompile 'junit:junit:4.10'
testCompile 'org.robolectric:robolectric:2.3-SNAPSHOT'
testCompile 'com.squareup:fest-android:1.0.+'
testCompile 'org.mockito:mockito-all:1.9.5'
instrumentTestCompile 'junit:junit:4.10'
instrumentTestCompile 'org.robolectric:robolectric:2.3-SNAPSHOT'
instrumentTestCompile 'com.squareup:fest-android:1.0.+'
instrumentTestCompile 'org.mockito:mockito-all:1.9.5'

Now while running gradle I get the following output (Again a snapshot):

Failed to get resource: GET. [HTTP HTTP/1.1 403 Forbidden: https://repo.commonsware.com.s3.amazonaws.com/de/greenrobot/greendao/]
Failed to get resource: GET. [HTTP HTTP/1.1 403 Forbidden: https://repo.commonsware.com.s3.amazonaws.com/com/android/support/support-v4/maven-metadata.xml]
Failed to get resource: GET. [HTTP HTTP/1.1 403 Forbidden: https://repo.commonsware.com.s3.amazonaws.com/com/android/support/support-v4/]
Failed to get resource: GET. [HTTP HTTP/1.1 403 Forbidden: https://repo.commonsware.com.s3.amazonaws.com/com/squareup/picasso/picasso/maven-metadata.xml]
Failed to get resource: GET. [HTTP HTTP/1.1 403 Forbidden: https://repo.commonsware.com.s3.amazonaws.com/com/squareup/picasso/picasso/]
Failed to get resource: GET. [HTTP HTTP/1.1 403 Forbidden: https://repo.commonsware.com.s3.amazonaws.com/com/squareup/picasso/picasso/maven-metadata.xml]
Failed to get resource: GET. [HTTP HTTP/1.1 403 Forbidden: https://repo.commonsware.com.s3.amazonaws.com/com/squareup/picasso/picasso/]
Failed to get resource: GET. [HTTP HTTP/1.1 403 Forbidden: https://repo.commonsware.com.s3.amazonaws.com/com/android/support/appcompat-v7/maven-metadata.xml]
Failed to get resource: GET. [HTTP HTTP/1.1 403 Forbidden: https://repo.commonsware.com.s3.amazonaws.com/com/android/support/appcompat-v7/]
Failed to get resource: GET. [HTTP HTTP/1.1 403 Forbidden: https://repo.commonsware.com.s3.amazonaws.com/com/android/support/mediarouter-v7/maven-metadata.xml]
Failed to get resource: GET. [HTTP HTTP/1.1 403 Forbidden: https://repo.commonsware.com.s3.amazonaws.com/com/android/support/mediarouter-v7/]
Failed to get resource: GET. [HTTP HTTP/1.1 403 Forbidden: https://repo.commonsware.com.s3.amazonaws.com/com/android/support/support-v4/maven-metadata.xml]
Failed to get resource: GET. [HTTP HTTP/1.1 403 Forbidden: https://repo.commonsware.com.s3.amazonaws.com/com/android/support/support-v4/]
Failed to get resource: GET. [HTTP HTTP/1.1 403 Forbidden: https://repo.commonsware.com.s3.amazonaws.com/com/fasterxml/jackson/core/jackson-databind/maven-metadata.xml]
Failed to get resource: GET. [HTTP HTTP/1.1 403 Forbidden: https://repo.commonsware.com.s3.amazonaws.com/com/fasterxml/jackson/core/jackson-databind/]
Failed to get resource: GET. [HTTP HTTP/1.1 403 Forbidden: https://repo.commonsware.com.s3.amazonaws.com/com/fasterxml/jackson/core/jackson-core/maven-metadata.xml]
Failed to get resource: GET. [HTTP HTTP/1.1 403 Forbidden: https://repo.commonsware.com.s3.amazonaws.com/com/fasterxml/jackson/core/jackson-core/]
Failed to get resource: GET. [HTTP HTTP/1.1 403 Forbidden: https://repo.commonsware.com.s3.amazonaws.com/com/fasterxml/jackson/core/jackson-annotations/maven-metadata.xml]
Failed to get resource: GET. [HTTP HTTP/1.1 403 Forbidden: https://repo.commonsware.com.s3.amazonaws.com/com/fasterxml/jackson/core/jackson-annotations/]
Failed to get resource: GET. [HTTP HTTP/1.1 403 Forbidden: https://repo.commonsware.com.s3.amazonaws.com/com/squareup/okhttp/okhttp/maven-metadata.xml]
Failed to get resource: GET. [HTTP HTTP/1.1 403 Forbidden: https://repo.commonsware.com.s3.amazonaws.com/com/squareup/okhttp/okhttp/]
Failed to get resource: GET. [HTTP HTTP/1.1 403 Forbidden: https://repo.commonsware.com.s3.amazonaws.com/com/squareup/retrofit/retrofit/maven-metadata.xml]
Failed to get resource: GET. [HTTP HTTP/1.1 403 Forbidden: https://repo.commonsware.com.s3.amazonaws.com/com/squareup/retrofit/retrofit/]
Failed to get resource: GET. [HTTP HTTP/1.1 403 Forbidden: https://repo.commonsware.com.s3.amazonaws.com/de/greenrobot/greendao/maven-metadata.xml]
Failed to get resource: GET. [HTTP HTTP/1.1 403 Forbidden: https://repo.commonsware.com.s3.amazonaws.com/de/greenrobot/greendao/]
Failed to get resource: GET. [HTTP HTTP/1.1 403 Forbidden: https://repo.commonsware.com.s3.amazonaws.com/com/android/support/support-v4/maven-metadata.xml]
Failed to get resource: GET. [HTTP HTTP/1.1 403 Forbidden: https://repo.commonsware.com.s3.amazonaws.com/com/android/support/support-v4/]
Failed to get resource: GET. [HTTP HTTP/1.1 403 Forbidden: https://repo.commonsware.com.s3.amazonaws.com/com/squareup/picasso/picasso/maven-metadata.xml]
Failed to get resource: GET. [HTTP HTTP/1.1 403 Forbidden: https://repo.commonsware.com.s3.amazonaws.com/com/squareup/picasso/picasso/]
Failed to get resource: GET. [HTTP HTTP/1.1 403 Forbidden: https://repo.commonsware.com.s3.amazonaws.com/com/squareup/picasso/picasso/maven-metadata.xml]
Failed to get resource: GET. [HTTP HTTP/1.1 403 Forbidden: https://repo.commonsware.com.s3.amazonaws.com/com/squareup/picasso/picasso/]
Failed to get resource: GET. [HTTP HTTP/1.1 403 Forbidden: https://repo.commonsware.com.s3.amazonaws.com/com/android/support/appcompat-v7/maven-metadata.xml]
Failed to get resource: GET. [HTTP HTTP/1.1 403 Forbidden: https://repo.commonsware.com.s3.amazonaws.com/com/android/support/appcompat-v7/]
Failed to get resource: GET. [HTTP HTTP/1.1 403 Forbidden: https://repo.commonsware.com.s3.amazonaws.com/com/android/support/mediarouter-v7/maven-metadata.xml]
Failed to get resource: GET. [HTTP HTTP/1.1 403 Forbidden: https://repo.commonsware.com.s3.amazonaws.com/com/android/support/mediarouter-v7/]
Failed to get resource: GET. [HTTP HTTP/1.1 403 Forbidden: https://repo.commonsware.com.s3.amazonaws.com/com/android/support/support-v4/maven-metadata.xml]

How can I force the build to do this lookup only once a day or to skip lookups that failed last time or to take the first artifact that was found and skip all the other repos for artifacts that are already available?

Cœur
  • 37,241
  • 25
  • 195
  • 267
Janusz
  • 187,060
  • 113
  • 301
  • 369
  • 1
    "Maybe this occurs because one of the repositories returns an error for most of the dependencies" -- my repository is up on Amazon S3, so you will get a 403 instead of a 404 for file-not-found. That should not impact performance, though apparently it impacts verbose logging (or wherever you are getting the log lines from your question). – CommonsWare Feb 05 '14 at 15:26
  • 1
    Apparently it can't find some dependencies at all. It will stop after the first repo that contains a matching module, and it will only try to re-resolve snapshots and dynamic versions after 24 hours (that's the default). (PS: If you have a proxy, it's usually better to declare all external repos there, rather than in the build script.) – Peter Niederwieser Feb 05 '14 at 15:28
  • @PeterNiederwieser the proxy is only working in one of the offices at the moment so we need a way to not use the proxy in other locations. – Janusz Feb 05 '14 at 15:30
  • @CommonsWare Those log lines are from verbose logging yes. It seems that gradle retries to retrieve the artifacts from your server every time because the 403 result is not cached for your repository. The gradle documentation states that gradle will always retrieve all artifact informations from all repos and then choose the best one. Maybe it will check your repo again for a better version of the artifacts every build? – Janusz Feb 05 '14 at 15:31
  • "the proxy is only working in one of the offices at the moment so we need a way to not use the proxy in other locations" -- you could toggle its use via an environment variable, or local properties file, or something. "The gradle documentation states that gradle will always retrieve all artifact informations from all repos and then choose the best one" -- that runs counter to what Peter wrote, though it would fit your symptoms. "because the 403 result is not cached for your repository" -- if that's true, that's bad. – CommonsWare Feb 05 '14 at 15:34
  • I added the sacklist and the mergeadapter as jars and the build dropped from 2min to 18 seconds. It seems that the response of your repository definitely troubles gradles caching mechanism. – Janusz Feb 05 '14 at 15:38
  • @Janusz: Try it again now. I have adjusted the S3 bucket setting, so URLs like the ones in your logging should return 404 instead of 403. – CommonsWare Feb 05 '14 at 15:44
  • Nice :) First build after change 45 seconds. Second build 17 seconds. Thanks that did the trick. – Janusz Feb 05 '14 at 15:55
  • For those seeing this with regard to Fabric, see this answer: http://stackoverflow.com/a/27130652/291827 – Jason Robinson Feb 13 '15 at 05:18

1 Answers1

1

The moral of this story: make sure that your Maven repository returns 404 for anything that is not actually there.

In the specific case of hosting a repository up on Amazon S3, grant "Everyone" the "List" permission using the Amazon S3 Manager, or otherwise follow the instructions from this StackOverflow answer.

Community
  • 1
  • 1
CommonsWare
  • 986,068
  • 189
  • 2,389
  • 2,491
  • 1
    It would be nice to force gradle to make look only for certain artifacts in certain repos to fix this in the build file. For those cases where the repo owner is not so responsive. Thanks again. – Janusz Feb 05 '14 at 16:02