7

Currently, I am facing the issue that I am trying to set up the S4SDK jenkins pipeline behind a proxy.

For this I adjusted the configuration in the server.cfg accordingly and it also shows in the Jenkins server log, that this configuration is used:

Setting Jenkins network proxy to <IP of proxy>:<port of proxy> without using credentials. No proxy patterns: [*localhost, *127.0.0.1, *s4sdk-nexus]

However during execution of the pipeline, more specifically when executing the s4sdkQualityChecks at the end the checkHystrix.groovy script uses grapes to pull another dependency:

@Grab('com.xlson.groovycsv:groovycsv:1.1')

However this dependency cannot be resolved and the following stacktrace appears:

org.jenkinsci.plugins.workflow.cps.CpsCompilationErrorsException: startup failed:
General error during conversion: Error grabbing Grapes -- [unresolved dependency: com.xlson.groovycsv#groovycsv;1.1: not found]

java.lang.RuntimeException: Error grabbing Grapes -- [unresolved dependency: com.xlson.groovycsv#groovycsv;1.1: not found]
    at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
    at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
    at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
    at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
    at org.codehaus.groovy.reflection.CachedConstructor.invoke(CachedConstructor.java:83)
    at org.codehaus.groovy.reflection.CachedConstructor.doConstructorInvoke(CachedConstructor.java:77)
    at org.codehaus.groovy.runtime.callsite.ConstructorSite$ConstructorSiteNoUnwrap.callConstructor(ConstructorSite.java:84)
    at org.codehaus.groovy.runtime.callsite.AbstractCallSite.callConstructor(AbstractCallSite.java:247)
    at groovy.grape.GrapeIvy.getDependencies(GrapeIvy.groovy:424)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:498)
    at org.codehaus.groovy.reflection.CachedMethod.invoke(CachedMethod.java:93)
    at groovy.lang.MetaMethod.doMethodInvoke(MetaMethod.java:325)
    at groovy.lang.MetaClassImpl.invokeMethod(MetaClassImpl.java:1213)
    at groovy.lang.MetaClassImpl.invokeMethod(MetaClassImpl.java:1022)
    at org.codehaus.groovy.runtime.callsite.PogoMetaClassSite.callCurrent(PogoMetaClassSite.java:69)
    at groovy.grape.GrapeIvy.resolve(GrapeIvy.groovy:571)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:498)
    at org.codehaus.groovy.reflection.CachedMethod.invoke(CachedMethod.java:93)
    at groovy.lang.MetaMethod.doMethodInvoke(MetaMethod.java:325)
    at groovy.lang.MetaClassImpl.invokeMethod(MetaClassImpl.java:1213)
    at groovy.lang.MetaClassImpl.invokeMethod(MetaClassImpl.java:1022)
    at org.codehaus.groovy.runtime.callsite.PogoMetaClassSite.callCurrent(PogoMetaClassSite.java:69)
    at org.codehaus.groovy.runtime.callsite.AbstractCallSite.callCurrent(AbstractCallSite.java:190)
    at groovy.grape.GrapeIvy.resolve(GrapeIvy.groovy:538)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:498)
    at org.codehaus.groovy.reflection.CachedMethod.invoke(CachedMethod.java:93)
    at groovy.lang.MetaMethod.doMethodInvoke(MetaMethod.java:325)
    at groovy.lang.MetaClassImpl.invokeMethod(MetaClassImpl.java:1213)
    at groovy.lang.MetaClassImpl.invokeMethod(MetaClassImpl.java:1022)
    at org.codehaus.groovy.runtime.callsite.PogoMetaClassSite.callCurrent(PogoMetaClassSite.java:69)
    at org.codehaus.groovy.runtime.callsite.AbstractCallSite.callCurrent(AbstractCallSite.java:182)
    at groovy.grape.GrapeIvy.grab(GrapeIvy.groovy:256)
    at groovy.grape.Grape.grab(Grape.java:167)
    at groovy.grape.GrabAnnotationTransformation.visit(GrabAnnotationTransformation.java:378)
    at org.codehaus.groovy.transform.ASTTransformationVisitor$3.call(ASTTransformationVisitor.java:321)
    at org.codehaus.groovy.control.CompilationUnit.applyToSourceUnits(CompilationUnit.java:943)
    at org.codehaus.groovy.control.CompilationUnit.doPhaseOperation(CompilationUnit.java:605)
    at org.codehaus.groovy.control.CompilationUnit.processPhaseOperations(CompilationUnit.java:581)
    at org.codehaus.groovy.control.CompilationUnit.compile(CompilationUnit.java:558)
    at groovy.lang.GroovyClassLoader.doParseClass(GroovyClassLoader.java:298)
    at groovy.lang.GroovyClassLoader.parseClass(GroovyClassLoader.java:268)
    at groovy.lang.GroovyClassLoader.parseClass(GroovyClassLoader.java:254)
    at groovy.lang.GroovyClassLoader.recompile(GroovyClassLoader.java:761)
    at groovy.lang.GroovyClassLoader.loadClass(GroovyClassLoader.java:718)
    at groovy.lang.GroovyClassLoader.loadClass(GroovyClassLoader.java:787)
    at java.lang.ClassLoader.loadClass(ClassLoader.java:411)
    at org.jenkinsci.plugins.workflow.cps.CpsGroovyShell$TimingLoader.loadClass(CpsGroovyShell.java:161)
    at java.lang.ClassLoader.loadClass(ClassLoader.java:411)
    at groovy.lang.GroovyClassLoader.loadClass(GroovyClassLoader.java:677)
    at groovy.lang.GroovyClassLoader.loadClass(GroovyClassLoader.java:787)
    at groovy.lang.GroovyClassLoader.loadClass(GroovyClassLoader.java:775)
    at org.jenkinsci.plugins.workflow.cps.global.UserDefinedGlobalVariable.getValue(UserDefinedGlobalVariable.java:57)
    at org.jenkinsci.plugins.workflow.cps.CpsScript.invokeMethod(CpsScript.java:113)
    at org.codehaus.groovy.runtime.callsite.PogoMetaClassSite.call(PogoMetaClassSite.java:48)
    at org.codehaus.groovy.runtime.callsite.CallSiteArray.defaultCall(CallSiteArray.java:48)
    at org.codehaus.groovy.runtime.callsite.AbstractCallSite.call(AbstractCallSite.java:113)
    at com.cloudbees.groovy.cps.sandbox.DefaultInvoker.methodCall(DefaultInvoker.java:20)
    at com.cloudbees.groovy.cps.impl.ContinuationGroup.methodCall(ContinuationGroup.java:57)
    at com.cloudbees.groovy.cps.impl.FunctionCallBlock$ContinuationImpl.dispatchOrArg(FunctionCallBlock.java:109)
    at com.cloudbees.groovy.cps.impl.FunctionCallBlock$ContinuationImpl.fixName(FunctionCallBlock.java:77)
    at sun.reflect.GeneratedMethodAccessor259.invoke(Unknown Source)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:498)
    at com.cloudbees.groovy.cps.impl.ContinuationPtr$ContinuationImpl.receive(ContinuationPtr.java:72)
    at com.cloudbees.groovy.cps.impl.ConstantBlock.eval(ConstantBlock.java:21)
    at com.cloudbees.groovy.cps.Next.step(Next.java:83)
    at com.cloudbees.groovy.cps.Continuable$1.call(Continuable.java:174)
    at com.cloudbees.groovy.cps.Continuable$1.call(Continuable.java:163)
    at org.codehaus.groovy.runtime.GroovyCategorySupport$ThreadCategoryInfo.use(GroovyCategorySupport.java:129)
    at org.codehaus.groovy.runtime.GroovyCategorySupport.use(GroovyCategorySupport.java:268)
    at com.cloudbees.groovy.cps.Continuable.run0(Continuable.java:163)
    at org.jenkinsci.plugins.workflow.cps.SandboxContinuable.access$001(SandboxContinuable.java:18)
    at org.jenkinsci.plugins.workflow.cps.SandboxContinuable.run0(SandboxContinuable.java:51)
    at org.jenkinsci.plugins.workflow.cps.CpsThread.runNextChunk(CpsThread.java:186)
    at org.jenkinsci.plugins.workflow.cps.CpsThreadGroup.run(CpsThreadGroup.java:370)
    at org.jenkinsci.plugins.workflow.cps.CpsThreadGroup.access$200(CpsThreadGroup.java:93)
    at org.jenkinsci.plugins.workflow.cps.CpsThreadGroup$2.call(CpsThreadGroup.java:282)
    at org.jenkinsci.plugins.workflow.cps.CpsThreadGroup$2.call(CpsThreadGroup.java:270)
    at org.jenkinsci.plugins.workflow.cps.CpsVmExecutorService$2.call(CpsVmExecutorService.java:64)
    at java.util.concurrent.FutureTask.run(FutureTask.java:266)
    at hudson.remoting.SingleLaneExecutorService$1.run(SingleLaneExecutorService.java:131)
    at jenkins.util.ContextResettingExecutorService$1.run(ContextResettingExecutorService.java:28)
    at jenkins.security.ImpersonatingExecutorService$1.run(ImpersonatingExecutorService.java:59)
    at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
    at java.util.concurrent.FutureTask.run(FutureTask.java:266)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
    at java.lang.Thread.run(Thread.java:748)

The proxy itself is also configured for the maven execution within the pipeline so it should in principal work. Not sure if I missed another configuration option, but haven't found any so far. Any advice on this would be highly appreciated.

Update

Current failing step is now the checkServices.groovy step with the following log trace:

16:19:13  --- BEGIN LIBRARY STEP: checkServices.groovy ---
[Pipeline] httpRequest
16:19:13  HttpMethod: GET
16:19:13  URL: https://api.sap.com/odata/1.0/catalog.svc/ContentEntities.ContentPackages('SAPS4HANACloud')/Artifacts?$format=json&$select=Name
16:19:13  Sending request to url: https://api.sap.com/odata/1.0/catalog.svc/ContentEntities.ContentPackages('SAPS4HANACloud')/Artifacts?$format=json&$select=Name
16:19:13  Treating UnknownHostException(http://[PROXY_IP]: Name or service not known) as 404 Not Found
[Pipeline] error
[Pipeline] echo
16:19:13  ----------------------------------------------------------
16:19:13  --- ERROR OCCURRED IN LIBRARY STEP: checkServices
16:19:13  ----------------------------------------------------------
16:19:13  
16:19:13  FOLLOWING PARAMETERS WERE AVAILABLE TO THIS STEP:
16:19:13  ***
16:19:13  [script:Script1@6e29ff5b, nonErpDestinations:null, customODataServices:null]
16:19:13  ***
16:19:13  
16:19:13  ERROR WAS:
16:19:13  ***
16:19:13  hudson.AbortException: Failed to download the list of available services from API Business Hub (https://api.sap.com/). Please check if your Jenkins can reach this web resource.
16:19:13  Exception: java.lang.IllegalStateException: hudson.AbortException: Fail: the returned code 404 is not in the accepted range: [[100?399]]
16:19:13  ***
16:19:13  
16:19:13  
[Pipeline] echo
16:19:14  --- END LIBRARY STEP: checkServices.groovy ---
Tim L.
  • 224
  • 2
  • 11
  • 1
    Hi Tim, did you set the `x_java_opts="-Xmx1024m -Dhttp.proxyHost=YOURPROXY -Dhttp.proxyPortYOURPORT -Dhttps.proxyHost=YOURPROXY -Dhttps.proxyPort=YOURPORT -Dhttp.nonProxyHosts=localhost|*.corp"` in server.cfg? – floge77 Jun 20 '19 at 10:48
  • 2
    Besides that we found [this article](https://tcollignon.github.io/2017/07/10/How-To-Use-third-party-libraries-in-Jenkins-Pipeline.html) where they advise to create a grapeConfig.xml in $JENKINS_HOME/.grovvy/. To try that out you would have to login to the container with `docker exec -it s4sdk-jenkins-master bash` – floge77 Jun 20 '19 at 10:55
  • Hi, thanks for these hints! The x_javaopts option I actually missed, but even using that option did not help. – Tim L. Jun 20 '19 at 19:21
  • Regarding the grapeConfig.xml, I added it with the following properties (based on the documentation for Apache Ivy): http.proxyHost, http.proxyPort, https.proxyHost, https.proxyPort (via ``) However, that did not help either for whatever reason - but the file was definitely picked up, because I missed to define a resolver initially and it threw another error. Will have to check tomorrow, whether there might be another setting/configuration I should use within the grapeConfig.xml. – Tim L. Jun 20 '19 at 19:26
  • 1
    Besides of that since I am one of the maintainer I can tell you that we're working on removing this dependency from the step. – floge77 Jun 21 '19 at 07:57
  • 1
    We removed the @Grab from the checkHystrix step. As soon as v21 is is released you can consume that version and the issue is solved. – floge77 Jun 24 '19 at 15:05
  • Thanks a logt, that will probably help! Currently, I haven't found another way to specify a proxy for this build step. Is there already a branch available with the fix, so I might already try out that new version? Since we don't have a productive setup yet, this would help us check, that it would work in the end. – Tim L. Jun 25 '19 at 19:40

1 Answers1

2

Updated answer:

This issue should be resolved on the current master branch and on the next released version (v22, no release date yet). Those don't rely on @Grab anymore, thus removing the source of the issue.

Florian Wilhelm
  • 600
  • 4
  • 17
  • Hi Florian, thanks for the info and great that this was so quickly fixed! However, I tried to re-run my pipeline and it seems the pipeline is not updated. Even when setting up the complete guideline again and fixing the version to v21 in the Jenkinsfile of my project, it still tries to use Grapes to pull the dependencies. Is there any pipeline cache I am not aware of which I need to clean? – Tim L. Jun 28 '19 at 09:48
  • Hello Tim, I just checked and found another use of `@Grab` also for CSV Parsing. I'm sorry I did not check properly in advance. I'll keep you posted on that issue. – Florian Wilhelm Jun 29 '19 at 10:22
  • 1
    Hi Tim, we've updated the `master` branch to also remove that `@Grab`. Can you verify if this does resolve the issue you're facing? (So to be clear, instead of `v21`, just set `master` in your `Jenkinsfile`). If not, I think I'll need a little more info on the setup you have. Thanks and kind regards, Florian – Florian Wilhelm Jul 02 '19 at 06:56
  • Hi Florian, thanks a lot! We got a few more steps working since no more grapes dependencies were tried to be resolved. So this issue is indeed resolved. However, the pipeline now fails during the quality check because the api.sap.com is not available. I checked on the Jenkins host and I can access it there over the proxy, so my guess is that the s4sdkQualityChecks don't use the proxy set up in server.cfg either? Do you see any workaround or solution for this? – Tim L. Jul 03 '19 at 08:58
  • Too bad, I'm sorry we keep running into new issues. For validation purposes, I created a branch called `fetch-url-proxy`, would you mind testing if that resolves the issue? Just replace `master` with that branch name in the `Jenkinsfile` and start the pipeline again. – Florian Wilhelm Jul 03 '19 at 10:36
  • Now, it throws a different error: Treating UnknownHostException(http://: Name or service not known) as 404 Not Found. Not sure, but it seems the port is missing? – Tim L. Jul 03 '19 at 11:32
  • Seems like the same issue here https://stackoverflow.com/questions/46319408/jenkins-httprequest-plugin-404-not-found Have you configured the Proxy in the Jenkins plugin manager in "Advanced"? – Florian Wilhelm Jul 03 '19 at 13:14
  • Yes, it is configured in the Plugin Manager and if I validate the proxy under Advanced with the https://api.sap.com URL it also suceeds. However, in the pipeline it fails... – Tim L. Jul 03 '19 at 14:22
  • Hi Tim, can you provide a larger part of the log (with confidential information removed) around where it fails? As of now I don't know how to fix that. – Florian Wilhelm Jul 04 '19 at 12:27
  • Hi Florian, added the log for current failing step. Let me know if you require further information. – Tim L. Jul 05 '19 at 04:23
  • Hi Tim, not sure if that fixes it, but I’ve pushed a new version to the `fetch-url-proxy` branch, could you check if that changes anything? If not, I think we need to schedule a call to discuss the details of the infrastructure setup we’re facing here. – Florian Wilhelm Jul 15 '19 at 10:24