1

I am using Clair for Vulnerability checks in my harbor.

Services like Clair that have thousands of hosts continually hitting the hosting git server(https://git.launchpad.net/ubuntu-cve-tracker/) saturate the server, and so there are scaling measures in place that causes it to return a 503 error when too many clients are concurrently hitting it.

These are my errors in my Clair pod:

{"Event":"could not pull ubuntu-cve-tracker repository","Level":"error","Location":"ubuntu.go:174",
"Time":"2021-06-25 06:38:32.859806","error":"exit status 128",
"output":"Cloning into '.'...

fatal: unable to access '[https://git.launchpad.net/ubuntu-cve-tracker/':|https://git.launchpad.net/ubuntu-cve-tracker/%27:] 
The requested URL returned error: 503\n"} 

{"Event":"an error occured when fetching update","Level":"error","Location":"updater.go:246",
"Time":"2021-06-25 06:38:32.859934","error":"could not download requested resource","updater name":"ubuntu"}

panic: runtime error: slice bounds out of range goroutine 549 [running]: github.com/coreos/clair/ext/vulnsrc/rhel.toFeatureVersions(0xc000208390, 0x2, 0xc000246070, 0x1, 0x1, 0xc0001bc200, 0x1, 0x1, 0x0, 0x908f38, ...) /go/src/github.com/coreos/clair/ext/vulnsrc/rhel/rhel.go:292 +0xc3b github.com/coreos/clair/ext/vulnsrc/rhel.parseRHSA(0x7fcc0f4a24b0, 0xc00038c0f0, 0xc00038c0f0, 0x7fcc0f4a24b0, 0xc00038c0f0, 0x8e2708, 0x4) /go/src/github.com/coreos/clair/ext/vulnsrc/rhel/rhel.go:182 +0x1c8

As per https://bugs.launchpad.net/ubuntu-cve-tracker/+bug/1925337 this is a bug from the git server, and in that post, they are suggesting to get Clair to pull data from other sources instead which means an offline approach. So apart from the offline approach, is there any other way to decrease the number of hits to the git server for Vulnerability checks?

I have tried to control the number of hits to the git server, but nowhere have I found the configuration in Clair.

Does anyone have any idea how we can control the hits for Vulnerability checks or avoid restarts of my pod?

Also, I found schedule a scan(hourly, daily, or weekly) on my harbor UI, But how does scheduling the scan to say daily help?
Is it only at that point it will try to do the git clone to get the latest CVEs?

Anvesh Muppeda
  • 95
  • 2
  • 6

1 Answers1

1

Check first if this is linked to goharbor/harbor issue 14720: "clair restarts repeatedly when there is some issue with vulnerability repos", with logs like

{"Event":"Start fetching vulnerabilities","Level":"info","Location":"ubuntu.go:85","Time":"2021-04-21 19:18:24.446743","package":"Ubuntu"}
...
{"Event":"could not pull ubuntu-cve-tracker repository","Level":"error","Location":"ubuntu.go:174","Time":"2021-04-21 19:18:25.147515","error":"exit status 128","output":"Cloning into '.'...\nfatal: unable to access 'https://git.launchpad.net/ubuntu-cve-tracker/': The requested URL returned error: 503\n"}
{"Event":"an error occured when fetching update","Level":"error","Location":"updater.go:246","Time":"2021-04-21 19:18:25.147607","error":"could not download requested resource","updater name":"ubuntu"}
...
panic: runtime error: slice bounds out of range [25:24]

goroutine 327 [running]:
github.com/quay/clair/v2/ext/vulnsrc/rhel.toFeatureVersions(0xc0065215a8, 0x2, 0xc0000b4f08, 0x1, 0x1, 0xc006ef7aa0, 0x1, 0x1, 0x2, 0xc0000b4ef0, ...)
    /go/src/github.com/quay/clair/ext/vulnsrc/rhel/rhel.go:276 +0xbf8

It refers to quay/clair issue 1249, but the harbor case is closed with PR 15032, using CLAIRVERSION=v2.1.7

VonC
  • 1,262,500
  • 529
  • 4,410
  • 5,250
  • Thank you @VonC for your answer and time. Yes, it is linked to https://github.com/goharbor/harbor/issues/14720 "clair restarts repeatedly when there is some issue with vulnerability repos". But as per https://bugs.launchpad.net/ubuntu-cve-tracker/+bug/1925337 this is a bug from the git server, and in that post, they are suggesting to get Clair to pull data from other sources instead which means an offline approach.  So apart from the offline approach, is there any other way to decrease the number of hits to the git server for Vulnerability checks? – Anvesh Muppeda Jul 21 '21 at 06:24
  • Also currently I am using below versions: Harbor: v1.9.3 clair-photon:v2.1.0 docker engine version: [19.03.11] By bumping up the harbor/Clair version fix my problem? docker run --name clair-db -d -p 5432:5432 arminc/clair-db:latest docker run --name clair-server -d --link clair-db:postgres -p 6060-6061 arminc/clair-local-scan:latest Clair version/image: quay.io/coreos/clair:latest Via: https://hub.docker.com/r/arminc/clair-local-scan tag=latest which was built approx 2021-04-18 – Anvesh Muppeda Jul 21 '21 at 06:24
  • panic: runtime error: slice bounds out of range goroutine 549 [running]: github.com/coreos/clair/ext/vulnsrc/rhel.toFeatureVersions(0xc000208390, 0x2, 0xc000246070, 0x1, 0x1, 0xc0001bc200, 0x1, 0x1, 0x0, 0x908f38, ...) /go/src/github.com/coreos/clair/ext/vulnsrc/rhel/rhel.go:292 +0xc3b github.com/coreos/clair/ext/vulnsrc/rhel.parseRHSA(0x7fcc0f4a24b0, 0xc00038c0f0, 0xc00038c0f0, 0x7fcc0f4a24b0, 0xc00038c0f0, 0x8e2708, 0x4) /go/src/github.com/coreos/clair/ext/vulnsrc/rhel/rhel.go:182 +0x1c8 – Anvesh Muppeda Jul 21 '21 at 06:28
  • @AnveshMuppeda I have not found another way to decrease the number of hits to the git server for Vulnerability checks, but do edit your question with the information from your previous comments: others will be able to review those. – VonC Jul 21 '21 at 06:45
  • Sure @VonC, thanks! Let me know if you are aware of bumping up the Harbor/Clair version to my fix the issue. – Anvesh Muppeda Jul 21 '21 at 06:59
  • @AnveshMuppeda I would bump the Harbor/clair version, at least for testing. – VonC Jul 21 '21 at 07:02