Multi-threading ant signjar occasionally causes errors

Question

My Gradle build signs almost 200 jars via the ant.signjar task. To speed it I decided to try and multi-thread it using info from here and here. FYI: This drops from 25 mins to 7 mins on my Core Duo laptop (4x faster!).

However, sometimes it runs successfully and sometimes it fails and I can't work out why. It is probably something to do with the multi-threading but I can't explain why it doesn't happen every time.

My code now looks like this.

import groovy.io.FileType
import static groovyx.gpars.GParsPool.withPool

//This is so that we can use the Groovy parallel library in the build script
buildscript {
  repositories {
    mavenLocal()
    mavenCentral()
    //Change to this if you have your own repo
    maven {
      url "http://localhost:8081/nexus/content/groups/public/"
    }
  }
  dependencies {
    classpath group: 'org.codehaus.gpars', name: 'gpars', version: '1.2.1'
  }
}

task signWebstartJars() << {
  File unsignedFolder = new File('build/maven')
  File signedFolder = new File('signed-libs')
  signedFolder.mkdirs()

  def list = []
  unsignedFolder.eachFileRecurse(FileType.FILES) { file ->
    list << file
  }

  list.sort()

  //Create a closure that allows us to run the actual ant task multi-threaded later
  def antClosure = {
    println "Updating manifest: " + it

    //Manifest needs extra attributes added for security
    ant.jar(destfile: it, update: true) {
      delegate.manifest {
        attribute(name: 'permissions', value: 'all-permissions')
        attribute(name: 'codebase', value: '*')
      }
    }

    println "Signing the jar: " + it
    ant.signjar(
      destDir: signedFolder,
      alias: alias,
      sigfile: sigfile,
      jar: it,
      keystore: keystore,
      storepass: storepass,
      storetype: storetype,
      preservelastmodified: "true",
      verbose: "true"
    )
  }

  //Use twice as many CPUs for no. of threads
  result = withPool(Runtime.runtime.availableProcessors() * 2) {
    list.collectParallel antClosure
  }
}

The error that sometimes occurs is this.

* What went wrong:
Execution failed for task ':signWebstartJars'.
> : manifest doesn't support the nested "jar" element.

* Try:
Run with --debug option to get more log output.

* Exception is:
org.gradle.api.tasks.TaskExecutionException: Execution failed for task ':signWebstartJars'.
    at org.gradle.api.internal.tasks.execution.ExecuteActionsTaskExecuter.executeActions(ExecuteActionsTaskExecuter.java:69)
    at org.gradle.api.internal.tasks.execution.ExecuteActionsTaskExecuter.execute(ExecuteActionsTaskExecuter.java:46)
    at org.gradle.api.internal.tasks.execution.PostExecutionAnalysisTaskExecuter.execute(PostExecutionAnalysisTaskExecuter.java:35)
    at org.gradle.api.internal.tasks.execution.SkipUpToDateTaskExecuter.execute(SkipUpToDateTaskExecuter.java:64)
    at org.gradle.api.internal.tasks.execution.ValidatingTaskExecuter.execute(ValidatingTaskExecuter.java:58)
    at org.gradle.api.internal.tasks.execution.SkipEmptySourceFilesTaskExecuter.execute(SkipEmptySourceFilesTaskExecuter.java:42)
    at org.gradle.api.internal.tasks.execution.SkipTaskWithNoActionsExecuter.execute(SkipTaskWithNoActionsExecuter.java:52)
    at org.gradle.api.internal.tasks.execution.SkipOnlyIfTaskExecuter.execute(SkipOnlyIfTaskExecuter.java:53)
    at org.gradle.api.internal.tasks.execution.ExecuteAtMostOnceTaskExecuter.execute(ExecuteAtMostOnceTaskExecuter.java:43)
    at org.gradle.api.internal.AbstractTask.executeWithoutThrowingTaskFailure(AbstractTask.java:296)
    at org.gradle.execution.taskgraph.AbstractTaskPlanExecutor$TaskExecutorWorker.executeTask(AbstractTaskPlanExecutor.java:79)
    at org.gradle.execution.taskgraph.AbstractTaskPlanExecutor$TaskExecutorWorker.processTask(AbstractTaskPlanExecutor.java:63)
    at org.gradle.execution.taskgraph.AbstractTaskPlanExecutor$TaskExecutorWorker.run(AbstractTaskPlanExecutor.java:51)
    at org.gradle.execution.taskgraph.DefaultTaskPlanExecutor.process(DefaultTaskPlanExecutor.java:23)
    at org.gradle.execution.taskgraph.DefaultTaskGraphExecuter.execute(DefaultTaskGraphExecuter.java:86)
    at org.gradle.execution.SelectedTaskExecutionAction.execute(SelectedTaskExecutionAction.java:29)
    at org.gradle.execution.DefaultBuildExecuter.execute(DefaultBuildExecuter.java:61)
    at org.gradle.execution.DefaultBuildExecuter.access$200(DefaultBuildExecuter.java:23)
    at org.gradle.execution.DefaultBuildExecuter$2.proceed(DefaultBuildExecuter.java:67)
    at org.gradle.execution.DryRunBuildExecutionAction.execute(DryRunBuildExecutionAction.java:32)
    at org.gradle.execution.DefaultBuildExecuter.execute(DefaultBuildExecuter.java:61)
    at org.gradle.execution.DefaultBuildExecuter.execute(DefaultBuildExecuter.java:54)
    at org.gradle.initialization.DefaultGradleLauncher.doBuildStages(DefaultGradleLauncher.java:148)
    at org.gradle.initialization.DefaultGradleLauncher.doBuild(DefaultGradleLauncher.java:105)
    at org.gradle.initialization.DefaultGradleLauncher.run(DefaultGradleLauncher.java:85)
    at org.gradle.launcher.exec.InProcessBuildActionExecuter$DefaultBuildController.run(InProcessBuildActionExecuter.java:81)
    at org.gradle.launcher.cli.ExecuteBuildAction.run(ExecuteBuildAction.java:33)
    at org.gradle.launcher.cli.ExecuteBuildAction.run(ExecuteBuildAction.java:24)
    at org.gradle.launcher.exec.InProcessBuildActionExecuter.execute(InProcessBuildActionExecuter.java:39)
    at org.gradle.launcher.daemon.server.exec.ExecuteBuild.doBuild(ExecuteBuild.java:45)
    at org.gradle.launcher.daemon.server.exec.BuildCommandOnly.execute(BuildCommandOnly.java:34)
    at org.gradle.launcher.daemon.server.exec.DaemonCommandExecution.proceed(DaemonCommandExecution.java:125)
    at org.gradle.launcher.daemon.server.exec.WatchForDisconnection.execute(WatchForDisconnection.java:35)
    at org.gradle.launcher.daemon.server.exec.DaemonCommandExecution.proceed(DaemonCommandExecution.java:125)
    at org.gradle.launcher.daemon.server.exec.ResetDeprecationLogger.execute(ResetDeprecationLogger.java:24)
    at org.gradle.launcher.daemon.server.exec.DaemonCommandExecution.proceed(DaemonCommandExecution.java:125)
    at org.gradle.launcher.daemon.server.exec.StartStopIfBuildAndStop.execute(StartStopIfBuildAndStop.java:33)
    at org.gradle.launcher.daemon.server.exec.DaemonCommandExecution.proceed(DaemonCommandExecution.java:125)
    at org.gradle.launcher.daemon.server.exec.ReturnResult.execute(ReturnResult.java:34)
    at org.gradle.launcher.daemon.server.exec.DaemonCommandExecution.proceed(DaemonCommandExecution.java:125)
    at org.gradle.launcher.daemon.server.exec.ForwardClientInput$2.call(ForwardClientInput.java:71)
    at org.gradle.launcher.daemon.server.exec.ForwardClientInput$2.call(ForwardClientInput.java:69)
    at org.gradle.util.Swapper.swap(Swapper.java:38)
    at org.gradle.launcher.daemon.server.exec.ForwardClientInput.execute(ForwardClientInput.java:69)
    at org.gradle.launcher.daemon.server.exec.DaemonCommandExecution.proceed(DaemonCommandExecution.java:125)
    at org.gradle.launcher.daemon.server.exec.LogToClient.doBuild(LogToClient.java:60)
    at org.gradle.launcher.daemon.server.exec.BuildCommandOnly.execute(BuildCommandOnly.java:34)
    at org.gradle.launcher.daemon.server.exec.DaemonCommandExecution.proceed(DaemonCommandExecution.java:125)
    at org.gradle.launcher.daemon.server.exec.EstablishBuildEnvironment.doBuild(EstablishBuildEnvironment.java:60)
    at org.gradle.launcher.daemon.server.exec.BuildCommandOnly.execute(BuildCommandOnly.java:34)
    at org.gradle.launcher.daemon.server.exec.DaemonCommandExecution.proceed(DaemonCommandExecution.java:125)
    at org.gradle.launcher.daemon.server.exec.StartBuildOrRespondWithBusy$1.run(StartBuildOrRespondWithBusy.java:45)
    at org.gradle.launcher.daemon.server.DaemonStateCoordinator.runCommand(DaemonStateCoordinator.java:184)
    at org.gradle.launcher.daemon.server.exec.StartBuildOrRespondWithBusy.doBuild(StartBuildOrRespondWithBusy.java:49)
    at org.gradle.launcher.daemon.server.exec.BuildCommandOnly.execute(BuildCommandOnly.java:34)
    at org.gradle.launcher.daemon.server.exec.DaemonCommandExecution.proceed(DaemonCommandExecution.java:125)
    at org.gradle.launcher.daemon.server.exec.HandleStop.execute(HandleStop.java:30)
    at org.gradle.launcher.daemon.server.exec.DaemonCommandExecution.proceed(DaemonCommandExecution.java:125)
    at org.gradle.launcher.daemon.server.exec.DaemonHygieneAction.execute(DaemonHygieneAction.java:39)
    at org.gradle.launcher.daemon.server.exec.DaemonCommandExecution.proceed(DaemonCommandExecution.java:125)
    at org.gradle.launcher.daemon.server.exec.CatchAndForwardDaemonFailure.execute(CatchAndForwardDaemonFailure.java:32)
    at org.gradle.launcher.daemon.server.exec.DaemonCommandExecution.proceed(DaemonCommandExecution.java:125)
    at org.gradle.launcher.daemon.server.exec.DefaultDaemonCommandExecuter.executeCommand(DefaultDaemonCommandExecuter.java:51)
    at org.gradle.launcher.daemon.server.DefaultIncomingConnectionHandler$ConnectionWorker.handleCommand(DefaultIncomingConnectionHandler.java:155)
    at org.gradle.launcher.daemon.server.DefaultIncomingConnectionHandler$ConnectionWorker.receiveAndHandleCommand(DefaultIncomingConnectionHandler.java:128)
    at org.gradle.launcher.daemon.server.DefaultIncomingConnectionHandler$ConnectionWorker.run(DefaultIncomingConnectionHandler.java:116)
    at org.gradle.internal.concurrent.DefaultExecutorFactory$StoppableExecutorImpl$1.run(DefaultExecutorFactory.java:64)
Caused by: : manifest doesn't support the nested "jar" element.
    at jsr166y.ForkJoinTask.getThrowableException(ForkJoinTask.java:504)
    at jsr166y.ForkJoinTask.reportResult(ForkJoinTask.java:567)
    at jsr166y.ForkJoinTask.join(ForkJoinTask.java:611)
    at jsr166y.ForkJoinPool.invoke(ForkJoinPool.java:1492)
    at groovyx.gpars.extra166y.AbstractParallelAnyArray.allObjects(AbstractParallelAnyArray.java:306)
    at groovyx.gpars.extra166y.ParallelArrayWithMapping.all(ParallelArrayWithMapping.java:141)
    at groovyx.gpars.GParsPoolUtil.collectParallel(GParsPoolUtil.java:495)
    at setup_g7k3p5ae0k79s0m3i6ril71p5$_run_closure10_closure45.doCall(D:\Calypso\clients\v14Build\setup.gradle:863)
    at groovyx.gpars.GParsPool$_withExistingPool_closure1.doCall(GParsPool.groovy:174)
    at groovyx.gpars.GParsPool$_withExistingPool_closure1.doCall(GParsPool.groovy)
    at groovyx.gpars.GParsPool.withExistingPool(GParsPool.groovy:173)
    at groovyx.gpars.GParsPool$withExistingPool$1.callStatic(Unknown Source)
    at groovyx.gpars.GParsPool$withExistingPool$1.callStatic(Unknown Source)
    at groovyx.gpars.GParsPool.withPool(GParsPool.groovy:145)
    at groovyx.gpars.GParsPool$withPool$0.callStatic(Unknown Source)
    at groovyx.gpars.GParsPool$withPool$0.callStatic(Unknown Source)
    at groovyx.gpars.GParsPool.withPool(GParsPool.groovy:121)
    at groovyx.gpars.GParsPool$withPool.callStatic(Unknown Source)
    at setup_g7k3p5ae0k79s0m3i6ril71p5$_run_closure10.doCall(D:\Calypso\clients\v14Build\setup.gradle:862)
    at org.gradle.api.internal.AbstractTask$ClosureTaskAction.execute(AbstractTask.java:539)
    at org.gradle.api.internal.AbstractTask$ClosureTaskAction.execute(AbstractTask.java:520)
    at org.gradle.api.internal.tasks.TaskMutator$1.execute(TaskMutator.java:77)
    at org.gradle.api.internal.tasks.TaskMutator$1.execute(TaskMutator.java:73)
    at org.gradle.api.internal.tasks.execution.ExecuteActionsTaskExecuter.executeAction(ExecuteActionsTaskExecuter.java:80)
    at org.gradle.api.internal.tasks.execution.ExecuteActionsTaskExecuter.executeActions(ExecuteActionsTaskExecuter.java:61)
    ... 66 more
Caused by: : manifest doesn't support the nested "jar" element.
    at jsr166y.ForkJoinTask.getThrowableException(ForkJoinTask.java:504)
    at jsr166y.ForkJoinTask.reportResult(ForkJoinTask.java:567)
    at jsr166y.ForkJoinTask.join(ForkJoinTask.java:611)
    at groovyx.gpars.extra166y.PAS$FJBase.internalCompute(PAS.java:120)
    at groovyx.gpars.extra166y.PAS$FJBase.compute(PAS.java:106)
    at jsr166y.RecursiveAction.exec(RecursiveAction.java:148)
    at jsr166y.ForkJoinTask.doExec(ForkJoinTask.java:305)
    at jsr166y.ForkJoinWorkerThread.execTask(ForkJoinWorkerThread.java:575)
    at jsr166y.ForkJoinPool.scan(ForkJoinPool.java:755)
    at jsr166y.ForkJoinPool.work(ForkJoinPool.java:617)
    at jsr166y.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:369)
Caused by: : manifest doesn't support the nested "jar" element.
    at org.apache.tools.ant.UnknownElement.handleChildren(UnknownElement.java:371)
    at org.apache.tools.ant.UnknownElement.handleChild(UnknownElement.java:593)
    at org.apache.tools.ant.UnknownElement.handleChildren(UnknownElement.java:358)
    at org.apache.tools.ant.UnknownElement.configure(UnknownElement.java:204)
    at org.apache.tools.ant.UnknownElement.maybeConfigure(UnknownElement.java:163)
    at org.gradle.api.internal.project.ant.BasicAntBuilder.nodeCompleted(BasicAntBuilder.java:71)
    at org.gradle.api.internal.project.ant.BasicAntBuilder.doInvokeMethod(BasicAntBuilder.java:86)
    at setup_g7k3p5ae0k79s0m3i6ril71p5$_run_closure10_closure44.doCall(D:\Calypso\clients\v14Build\setup.gradle:840)
    at groovyx.gpars.pa.CallClosure.call(CallClosure.java:47)
    at groovyx.gpars.pa.ClosureMapper.op(ClosureMapper.java:36)
    at groovyx.gpars.extra166y.AbstractParallelAnyArray$OOMPap.leafTransfer(AbstractParallelAnyArray.java:2255)
    at groovyx.gpars.extra166y.PAS$FJOMap.atLeaf(PAS.java:258)
    at groovyx.gpars.extra166y.PAS$FJBase.internalCompute(PAS.java:120)
    at groovyx.gpars.extra166y.PAS$FJBase.compute(PAS.java:106)
    at jsr166y.RecursiveAction.exec(RecursiveAction.java:148)
    at jsr166y.ForkJoinTask.doExec(ForkJoinTask.java:305)
    at jsr166y.ForkJoinWorkerThread.execTask(ForkJoinWorkerThread.java:575)
    at jsr166y.ForkJoinPool.scan(ForkJoinPool.java:733)
    ... 2 more
Caused by: class org.apache.tools.ant.taskdefs.Manifest doesn't support the nested "jar" element.
    at org.apache.tools.ant.IntrospectionHelper.throwNotSupported(IntrospectionHelper.java:509)
    at org.apache.tools.ant.UnknownElement.handleChildren(UnknownElement.java:361)
    ... 19 more

Are these JARs also being build with Gradle? Any reason you can't sign them as part of their build process rather than doing them in batch as you're doing now? — Mark Vieira, Mar 14 '15 at 02:39
These are third-party jars that I need to sign with my own certificate in order to launch my application with Java Web Start. — opticyclic, Mar 14 '15 at 21:26

score 1 · Accepted Answer · answered Mar 15 '15 at 05:17

It seems like AntBuilder isn't thread safe. Creating a new AntBuilder in the closure fixes it. I have pasted the snippet below from the Gradle Forum as it is shorter than my example in the original post.

import groovyx.gpars.GParsPool
buildscript {
    repositories {
        mavenCentral()
    }
    dependencies {
        classpath group: 'org.codehaus.gpars', name: 'gpars', version: '1.1.0'
    }
}
task signMyJars {
    doLast {
        List filesToSign = [
            file('file1.jar'),file('file2.jar'),file('file3.jar')
        ]
        GParsPool.withPool {
            filesToSign.eachParallel { f ->
                def antLocal = project.createAntBuilder()
                antLocal.signjar(
                    alias: "my key alias",
                    jar: f,
                    keystore: "my_key_store",
                    storepass: "key_store_pass",
                    storetype: "pkcs12",
                )
            }
        }
    }
}

Multi-threading ant signjar occasionally causes errors

1 Answers1