We run a multi-node multi-master-jenkins setup with a project that triggers a project on another jenkins instance via a cURL call.
Job A on Jenkins Alpha (all CentOS6/7/8) calls Job B on Jenkins Beta (CentOS6) like this:
curl -v -k --noproxy awesomehost.awesome.net -X POST https://usernick:API_TOKEN@awesomehost.awesome.net:8443/job/Example_XY/build -F file0=@${WORKSPACE}/beautifulzip.zip -F json='{"parameter": [{"name":"myinputzip.zip", "file":"file0"}]}'
This triggering job is run on multiple nodes and when using https:// that call fails with
Initializing NSS with certpath: sql:/etc/pki/nssdb
* skipping SSL peer certificate verification
* SSL connection using TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384
* Server certificate:
* subject: CN=awesomehost.awesome.net,OU=redacted,O=company,C=yeawhat
* start date: Mar 03 10:05:01 2021 GMT
* expire date: Mar 03 10:05:01 2022 GMT
* common name: awesomehost.awesome.net
* issuer: CN=nothingtoseehere,OU=movealong,O=evilcorp,L=raccooncity,ST=solid,C=yeawhat
* Server auth using Basic with user 'nick'
> POST /job/Example_XY/build HTTP/1.1
> Authorization: Basic cut==
> User-Agent: curl/7.29.0
> Host: awesomehost.awesome.net:8443
> Accept: */*
> Content-Length: 12479660
> Expect: 100-continue
> Content-Type: multipart/form-data; boundary=----------------------------34737a99beef
>
< HTTP/1.1 100 Continue
} [data not shown]
* SSL write: error -5961 (PR_CONNECT_RESET_ERROR)
* TCP connection reset by peer
Now, if I run the same cURL as http://, it works every time, but using https:// results in a failure most of the time. So it's most likely an HTTPS issue (wild guess).
But: while trying to debug, I used --trace and mysteriously everything works. every. time. Trace-time is not sufficient, but --trace - fixed the issue.
curl -v -k --trace - --noproxy awesomehost.awesome.net -X POST https://usernick:API_TOKEN@awesomehost.awesome.net:8443/job/Example_XY/build -F file0=@${WORKSPACE}/beautifulzip.zip -F json='{"parameter": [{"name":"myinputzip.zip", "file":"file0"}]}'
doesn't show the same error. Presuming some I/O related issue (all the systems share a nfs exported setup). I was curious if logfile I/O was the culprit, but running:
curl -v -k --trace - --noproxy awesomehost.awesome.net -X POST https://usernick:API_TOKEN@awesomehost.awesome.net:8443/job/Example_XY/build -F file0=@${WORKSPACE}/beautifulzip.zip -F json='{"parameter": [{"name":"myinputzip.zip", "file":"file0"}]}' 1>/dev/null
also works every time. Writing a long logfile doesn't seem to be the issue. Maybe some race condition?
Now, I don't have a real problem, as I have two ways to get stuff to work, but fixing stuff by turning on debug feels like a cheat.
cURL SSL connect error 35 with NSS error -5961 doesn't really seem to apply, as turning on debug fixes my issue.
Does anyone have a good idea how to debug the issue further? I can't promise that I can try out everything, as I am limited to non-root access. I would have to convince the - rightfully paranoid - admins to let me tinker with their farm, which I would rather not do, as my Jenkins is not the most important part of software running there.
Any ideas?
