1

Jenkins keeps failing to build my job because of what appears to be an SVN error (see below; I replaced the URLs and job name for security purposes). There's been other posts about this error but none of them seem to help. Also, I noticed that the workspace is not copying all the necessary files over from the SVN repository to the workspace. In fact, it's only fetching the python virtual environment folder and a couple others and none of the source code files.

Sorry about formatting, this is my first post on here. Thank you for the help.

Workspace appear to be locked, so getting a fresh workspace

Cleaning local Directory.

Checking out job at revision '2022-09-13T10:46:48.385 -0400' --quiet

Using sole credentials Subversion Credential in realm ‘url Subversion Repository’

ERROR: Failed to check out job

org.tmatesoft.svn.core.SVNException: svn: E175002: Connection reset
svn: E175002: REPORT request failed on '.../!svn/vcc/default'
    at org.tmatesoft.svn.core.internal.wc.SVNErrorManager.error(SVNErrorManager.java:112)
    at org.tmatesoft.svn.core.internal.wc.SVNErrorManager.error(SVNErrorManager.java:96)
    at org.tmatesoft.svn.core.internal.io.dav.http.HTTPConnection.request(HTTPConnection.java:765)
    at org.tmatesoft.svn.core.internal.io.dav.http.HTTPConnection.request(HTTPConnection.java:352)
    at org.tmatesoft.svn.core.internal.io.dav.http.HTTPConnection.request(HTTPConnection.java:340)
    at org.tmatesoft.svn.core.internal.io.dav.DAVConnection.performHttpRequest(DAVConnection.java:910)
    at org.tmatesoft.svn.core.internal.io.dav.DAVConnection.doReport(DAVConnection.java:363)
    at org.tmatesoft.svn.core.internal.io.dav.DAVRepository.runReport(DAVRepository.java:1361)
    at org.tmatesoft.svn.core.internal.io.dav.DAVRepository.update(DAVRepository.java:859)
    at org.tmatesoft.svn.core.internal.wc16.SVNUpdateClient16.update(SVNUpdateClient16.java:507)
    at org.tmatesoft.svn.core.internal.wc16.SVNUpdateClient16.doCheckout(SVNUpdateClient16.java:915)
    at org.tmatesoft.svn.core.internal.wc2.old.SvnOldCheckout.run(SvnOldCheckout.java:66)
    at org.tmatesoft.svn.core.internal.wc2.old.SvnOldCheckout.run(SvnOldCheckout.java:18)
    at org.tmatesoft.svn.core.internal.wc2.SvnOperationRunner.run(SvnOperationRunner.java:21)
    at org.tmatesoft.svn.core.wc2.SvnOperationFactory.run(SvnOperationFactory.java:1239)
    at org.tmatesoft.svn.core.wc2.SvnOperation.run(SvnOperation.java:294)
    at hudson.scm.subversion.CheckoutUpdater$SubversionUpdateTask.perform(CheckoutUpdater.java:133)
    at hudson.scm.subversion.WorkspaceUpdater$UpdateTask.delegateTo(WorkspaceUpdater.java:168)
    at hudson.scm.subversion.WorkspaceUpdater$UpdateTask.delegateTo(WorkspaceUpdater.java:176)
    at hudson.scm.subversion.UpdateUpdater$TaskImpl.perform(UpdateUpdater.java:185)
    at hudson.scm.subversion.WorkspaceUpdater$UpdateTask.delegateTo(WorkspaceUpdater.java:168)
    at hudson.scm.SubversionSCM$CheckOutTask.perform(SubversionSCM.java:1073)
    at hudson.scm.SubversionSCM$CheckOutTask.invoke(SubversionSCM.java:1049)
    at hudson.scm.SubversionSCM$CheckOutTask.invoke(SubversionSCM.java:1022)
    at hudson.FilePath$FileCallableWrapper.call(FilePath.java:3317)
    at hudson.remoting.UserRequest.perform(UserRequest.java:211)
    at hudson.remoting.UserRequest.perform(UserRequest.java:54)
    at hudson.remoting.Request$2.run(Request.java:376)
    at hudson.remoting.InterceptingExecutorService.lambda$wrap$0(InterceptingExecutorService.java:78)
    at java.util.concurrent.FutureTask.run(Unknown Source)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
    at hudson.remoting.Engine$1.lambda$newThread$0(Engine.java:119)
    at java.lang.Thread.run(Unknown Source)
Caused by: java.net.SocketException: Connection reset
    at java.net.SocketInputStream.read(Unknown Source)
    at java.net.SocketInputStream.read(Unknown Source)
    at java.io.BufferedInputStream.fill(Unknown Source)
    at java.io.BufferedInputStream.read1(Unknown Source)
    at java.io.BufferedInputStream.read(Unknown Source)
    at org.tmatesoft.svn.core.internal.util.ChunkedInputStream.read(ChunkedInputStream.java:75)
    at java.util.zip.InflaterInputStream.fill(Unknown Source)
    at java.util.zip.InflaterInputStream.read(Unknown Source)
    at java.util.zip.GZIPInputStream.read(Unknown Source)
    at sun.nio.cs.StreamDecoder.readBytes(Unknown Source)
    at sun.nio.cs.StreamDecoder.implRead(Unknown Source)
    at sun.nio.cs.StreamDecoder.read(Unknown Source)
    at java.io.InputStreamReader.read(Unknown Source)
    at org.tmatesoft.svn.core.internal.io.dav.http.XMLReader.read(XMLReader.java:39)
    at com.sun.org.apache.xerces.internal.impl.XMLEntityScanner.load(Unknown Source)
    at com.sun.org.apache.xerces.internal.impl.XMLEntityScanner.skipChar(Unknown Source)
    at com.sun.org.apache.xerces.internal.impl.XMLDocumentFragmentScannerImpl$FragmentContentDriver.next(Unknown Source)
    at com.sun.org.apache.xerces.internal.impl.XMLDocumentScannerImpl.next(Unknown Source)
    at com.sun.org.apache.xerces.internal.impl.XMLNSDocumentScannerImpl.next(Unknown Source)
    at com.sun.org.apache.xerces.internal.impl.XMLDocumentFragmentScannerImpl.scanDocument(Unknown Source)
    at com.sun.org.apache.xerces.internal.parsers.XML11Configuration.parse(Unknown Source)
    at com.sun.org.apache.xerces.internal.parsers.XML11Configuration.parse(Unknown Source)
    at com.sun.org.apache.xerces.internal.parsers.XMLParser.parse(Unknown Source)
    at com.sun.org.apache.xerces.internal.parsers.AbstractSAXParser.parse(Unknown Source)
    at com.sun.org.apache.xerces.internal.jaxp.SAXParserImpl$JAXPSAXParser.parse(Unknown Source)
    at org.tmatesoft.svn.core.internal.io.dav.http.HTTPConnection.readData(HTTPConnection.java:898)
    at org.tmatesoft.svn.core.internal.io.dav.http.HTTPConnection.readData(HTTPConnection.java:863)
    at org.tmatesoft.svn.core.internal.io.dav.http.HTTPRequest.dispatch(HTTPRequest.java:220)
    at org.tmatesoft.svn.core.internal.io.dav.http.HTTPConnection.request(HTTPConnection.java:451)
    ... 31 more
java.net.SocketException: Connection reset
    at java.net.SocketInputStream.read(Unknown Source)
    at java.net.SocketInputStream.read(Unknown Source)
    at java.io.BufferedInputStream.fill(Unknown Source)
    at java.io.BufferedInputStream.read1(Unknown Source)
    at java.io.BufferedInputStream.read(Unknown Source)
    at org.tmatesoft.svn.core.internal.util.ChunkedInputStream.read(ChunkedInputStream.java:75)
    at java.util.zip.InflaterInputStream.fill(Unknown Source)
    at java.util.zip.InflaterInputStream.read(Unknown Source)
    at java.util.zip.GZIPInputStream.read(Unknown Source)
    at sun.nio.cs.StreamDecoder.readBytes(Unknown Source)
    at sun.nio.cs.StreamDecoder.implRead(Unknown Source)
    at sun.nio.cs.StreamDecoder.read(Unknown Source)
    at java.io.InputStreamReader.read(Unknown Source)
    at org.tmatesoft.svn.core.internal.io.dav.http.XMLReader.read(XMLReader.java:39)
    at com.sun.org.apache.xerces.internal.impl.XMLEntityScanner.load(Unknown Source)
    at com.sun.org.apache.xerces.internal.impl.XMLEntityScanner.skipChar(Unknown Source)
    at com.sun.org.apache.xerces.internal.impl.XMLDocumentFragmentScannerImpl$FragmentContentDriver.next(Unknown Source)
    at com.sun.org.apache.xerces.internal.impl.XMLDocumentScannerImpl.next(Unknown Source)
    at com.sun.org.apache.xerces.internal.impl.XMLNSDocumentScannerImpl.next(Unknown Source)
    at com.sun.org.apache.xerces.internal.impl.XMLDocumentFragmentScannerImpl.scanDocument(Unknown Source)
    at com.sun.org.apache.xerces.internal.parsers.XML11Configuration.parse(Unknown Source)
    at com.sun.org.apache.xerces.internal.parsers.XML11Configuration.parse(Unknown Source)
    at com.sun.org.apache.xerces.internal.parsers.XMLParser.parse(Unknown Source)
    at com.sun.org.apache.xerces.internal.parsers.AbstractSAXParser.parse(Unknown Source)
    at com.sun.org.apache.xerces.internal.jaxp.SAXParserImpl$JAXPSAXParser.parse(Unknown Source)
    at org.tmatesoft.svn.core.internal.io.dav.http.HTTPConnection.readData(HTTPConnection.java:898)
    at org.tmatesoft.svn.core.internal.io.dav.http.HTTPConnection.readData(HTTPConnection.java:863)
    at org.tmatesoft.svn.core.internal.io.dav.http.HTTPRequest.dispatch(HTTPRequest.java:220)
    at org.tmatesoft.svn.core.internal.io.dav.http.HTTPConnection.request(HTTPConnection.java:451)
Caused: org.tmatesoft.svn.core.SVNException: svn: E175002: Connection reset
svn: E175002: REPORT request failed on '.../!svn/vcc/default'
    at org.tmatesoft.svn.core.internal.wc.SVNErrorManager.error(SVNErrorManager.java:112)
    at org.tmatesoft.svn.core.internal.wc.SVNErrorManager.error(SVNErrorManager.java:96)
    at org.tmatesoft.svn.core.internal.io.dav.http.HTTPConnection.request(HTTPConnection.java:765)
    at org.tmatesoft.svn.core.internal.io.dav.http.HTTPConnection.request(HTTPConnection.java:352)
    at org.tmatesoft.svn.core.internal.io.dav.http.HTTPConnection.request(HTTPConnection.java:340)
    at org.tmatesoft.svn.core.internal.io.dav.DAVConnection.performHttpRequest(DAVConnection.java:910)
    at org.tmatesoft.svn.core.internal.io.dav.DAVConnection.doReport(DAVConnection.java:363)
    at org.tmatesoft.svn.core.internal.io.dav.DAVRepository.runReport(DAVRepository.java:1361)
    at org.tmatesoft.svn.core.internal.io.dav.DAVRepository.update(DAVRepository.java:859)
    at org.tmatesoft.svn.core.internal.wc16.SVNUpdateClient16.update(SVNUpdateClient16.java:507)
    at org.tmatesoft.svn.core.internal.wc16.SVNUpdateClient16.doCheckout(SVNUpdateClient16.java:915)
    at org.tmatesoft.svn.core.internal.wc2.old.SvnOldCheckout.run(SvnOldCheckout.java:66)
    at org.tmatesoft.svn.core.internal.wc2.old.SvnOldCheckout.run(SvnOldCheckout.java:18)
    at org.tmatesoft.svn.core.internal.wc2.SvnOperationRunner.run(SvnOperationRunner.java:21)
    at org.tmatesoft.svn.core.wc2.SvnOperationFactory.run(SvnOperationFactory.java:1239)
    at org.tmatesoft.svn.core.wc2.SvnOperation.run(SvnOperation.java:294)
    at hudson.scm.subversion.CheckoutUpdater$SubversionUpdateTask.perform(CheckoutUpdater.java:133)
Retrying after 10 seconds.
Ben
  • 11
  • 3
  • Hi, we are facing exactly the same issue in our company. We are clueless. BTW this is a duplicate to so https://stackoverflow.com/questions/57996554/svn-checkout-sometimes-fails-in-jenkins it would be great to merge those. – m1lhaus Sep 14 '22 at 10:10
  • `Connection reset` indicates that a network problem could be the cause (or the server just dropped the connection). – Robert Sep 14 '22 at 12:35
  • @m1lhaus Glad I'm not the only one, let me know if you make any discoveries. – Ben Sep 14 '22 at 13:20
  • @m1lhaus By any chance, does your script call the jenkins-cli.jar? – Ben Sep 14 '22 at 13:25
  • @Ben No, we are not using jenkins-cli.jar – m1lhaus Sep 16 '22 at 07:56

2 Answers2

0

@m1haus I think I figured it out. The connection was timing out/resetting because it was fetching too many files from the repo. There was a lot because it was getting all of my python venv files. So I only committed the source code files to svn and not the venv files and it worked.

Ben
  • 11
  • 3
  • It is most likely linked to http timout set on jenkins server. In past days we did multiple changes on both jenkins and svn server side. I think the one helped the most was increasing http timeout on the server. Yesterday was the first time our stress test finished successfully. The stress tests was to checkout a 30GB repo in 4 paralel instances running like 8-9 hours. Now we are reverting back the changes on our test jenkins instance to find out which change specifically did it. Once I am back from vacation, I will then post official answer with complete info. – m1lhaus Sep 16 '22 at 07:59
  • @m1lhaus Yup, that makes sense. I look forward to hearing from you when you report back – Ben Sep 16 '22 at 20:15
0

We were experiencing the same behavior in our company. We are using latest VisualSVN Server v5. Whenever we checked-out huge amount of files (e.g. tons of tools), we were seeing exactly this non-informative error. Since we are Jenkins noobs, it was quite a struggle so I hope this helps.

Initially I tried suggested fix with -Dhudson.spool-svn=true, but that didn't help at all. Later on I went to see the source of Jenkins SVN client to check whether this flag is still applicable and used in the source. It is. But when browsing though the docs I realized that I am setting the flag wrong in Jenkins config file. Since when I went to Jenkins console and called System.getProperty('hudson.spool-svn'), it returned nothing. So I copied -Dhudson.spool-svn=true from JENKINS_JAVA_OPTIONS to JENKINS_ARGS, restarted the service and since then System.getProperty('hudson.spool-svn') returned value true. But that didn't solve the issue.

After updating JDK to v17 on both master and slave I could occasionally see a different error. Something about connection timeout. After some googling we increased http-timeout and situation improved. I could still see this issue when running a stress test (more parallel checkouts) after few hours.

Searching though VisualSVN page I discovered page talking about compression and caching. We disabled the compression as recommended for fast networks (1+GB/s in our case) and increased cache size (1GB I think). Since then all stress tests passed and even multiple paralel checkouts running 9 hours finished. After enabling the compression back on, the issue reappeared.

So I am pretty sure following steps are not solution for the root cause, but rather for its effects. I believe the root cause is combination of our network and not robust Jenkins SVN client

TL;DR

  1. Add -Dhudson.spool-svn=true to JENKINS_ARGS and to JENKINS_JAVA_OPTIONS under /etc/sysconfig/jenkins and restart the service

  2. Verify that hudson.spool-svn property was set by calling System.getProperty('hudson.spool-svn') under <your_jenkins_url>/script URL. It should return true

  3. Increase http-timeout on your SVN server, see e.g. https://www.visualsvn.com/support/topic/00170/

  4. In case of VisualSVN server, adjust cache size and disable compression, see https://www.visualsvn.com/support/topic/00114/

m1lhaus
  • 849
  • 1
  • 6
  • 10