Whitesource Bolt Azure DevOps plugin takes long time (and is too chatty)

Question

This happened since yesterday.

In the past months, I have integrated Whitesource Bolt scan (a free alternative to the popular Snyk) into our DevOps projects.

It normally took a couple of minutes to scan our packages, and we wew happy with the pipelines.

This is a typical redacted log from the pipelines

Starting: WhiteSource Bolt Scan
==============================================================================
Task         : WhiteSource Bolt
Description  : Detect security vulnerabilities, problematic open source licenses.
Version      : 21.3.2
Author       : WhiteSource
Help         : http://www.whitesourcesoftware.com
==============================================================================
Working directory is /home/vsts/work/1/s
Getting scan config data
unifiedAgent.config file created successfully at /home/vsts/work/1/s
Finished getScanConfigData
Finished archive and encryption
Starting Upload zip file to s3
Getting temp credentials
Finished to prepare scm scan request
Sending SCM scan request
Succeed to send SCM scan request
WhiteSource Support Token: 
Async Command Start: Add Build Tag
Build '4998' has following tags now: ws_support_token=ws_scan_start_time=Wed, 05 May 2021 12_32_26 GMT
Async Command End: Add Build Tag
Async Command Start: Add Build Tag
Build '4998' has following tags now: ws_support_token=
Async Command End: Add Build Tag
Finishing: WhiteSource Bolt Scan

Since yesterday, the output log exploded into the following endless debug logs, taking 30 minutes for an Angular project

Starting: WhiteSource Bolt Scan
==============================================================================
Task         : WhiteSource Bolt
Description  : Detect security vulnerabilities, problematic open source licenses.
Version      : 21.6.2
Author       : WhiteSource
Help         : http://www.whitesourcesoftware.com
==============================================================================





[CTX=sjgpwi107sx5to1j1pxbeasjdlvfsjqhzf6oqzeo1phtb]     resolved url in file = https://pkgs.dev.azure.com/_/_packaging/_/npm/registry/@babel/plugin-transform-template-literals/-/plugin-transform-template-literals-7.13.0.tgz
[DEBUG] [2021-07-06 08:41:49,836 +0000] - [CTX=sjgpwi107sx5to1j1pxbeasjdlvfsjqhzf6oqzeo1phtb]   resolved url in link = http://pkgs.dev.azure.com/@babel/plugin-transform-template-literals/7.13.0
[DEBUG] [2021-07-06 08:41:49,918 +0000] - [CTX=sjgpwi107sx5to1j1pxbeasjdlvfsjqhzf6oqzeo1phtb]   npm.accessToken is not defined
[DEBUG] [2021-07-06 08:41:50,043 +0000] - [CTX=sjgpwi107sx5to1j1pxbeasjdlvfsjqhzf6oqzeo1phtb]   Succeed to download the npm package @babel/plugin-transform-modules-umd-7.13.0.tgz-7.13.0.
[DEBUG] [2021-07-06 08:41:50,043 +0000] - [CTX=sjgpwi107sx5to1j1pxbeasjdlvfsjqhzf6oqzeo1phtb]   resolved url in file = https://pkgs.dev.azure.com/_/_packaging/_/npm/registry/@babel/plugin-transform-modules-amd/-/plugin-transform-modules-amd-7.13.0.tgz
[DEBUG] [2021-07-06 08:41:50,043 +0000] - [CTX=sjgpwi107sx5to1j1pxbeasjdlvfsjqhzf6oqzeo1phtb]   resolved url in link = http://pkgs.dev.azure.com/@babel/plugin-transform-modules-amd/7.13.0
[DEBUG] [2021-07-06 08:41:50,085 +0000] - [CTX=sjgpwi107sx5to1j1pxbeasjdlvfsjqhzf6oqzeo1phtb]   npm.accessToken is not defined
[DEBUG] [2021-07-06 08:41:50,085 +0000] - [CTX=sjgpwi107sx5to1j1pxbeasjdlvfsjqhzf6oqzeo1phtb]   Succeed to download the npm package @babel/plugin-syntax-optional-chaining-7.8.3.tgz-7.8.3.
[DEBUG] [2021-07-06 08:41:50,086 +0000] - [CTX=sjgpwi107sx5to1j1pxbeasjdlvfsjqhzf6oqzeo1phtb]   resolved url in file = https://pkgs.dev.azure.com/_/_packaging/_/npm/registry/babel-plugin-dynamic-import-node/-/babel-plugin-dynamic-import-node-2.3.3.tgz
[DEBUG] [2021-07-06 08:41:50,086 +0000] - [CTX=sjgpwi107sx5to1j1pxbeasjdlvfsjqhzf6oqzeo1phtb]   resolved url in link = http://pkgs.dev.azure.com/babel-plugin-dynamic-import-node/2.3.3
[DEBUG] [2021-07-06 08:41:50,146 +0000] - [CTX=sjgpwi107sx5to1j1pxbeasjdlvfsjqhzf6oqzeo1phtb]   npm.accessToken is not defined
[DEBUG] [2021-07-06 08:41:50,147 +0000] - [CTX=sjgpwi107sx5to1j1pxbeasjdlvfsjqhzf6oqzeo1phtb]   Succeed to download the npm package @babel/compat-data-7.13.8.tgz-7.13.8.
[DEBUG] [2021-07-06 08:41:50,147 +0000] - [CTX=sjgpwi107sx5to1j1pxbeasjdlvfsjqhzf6oqzeo1phtb]   resolved url in file = https://registry.npmjs.org/object.assign/-/object.assign-4.1.0.tgz
[DEBUG] [2021-07-06 08:41:50,147 +0000] - [CTX=sjgpwi107sx5to1j1pxbeasjdlvfsjqhzf6oqzeo1phtb]   resolved url in link = http://registry.npmjs.org/object.assign/4.1.0
[DEBUG] [2021-07-06 08:41:50,256 +0000] - [CTX=sjgpwi107sx5to1j1pxbeasjdlvfsjqhzf6oqzeo1phtb]   npm.accessToken is not defined
[DEBUG] [2021-07-06 08:41:50,258 +0000] - [CTX=sjgpwi107sx5to1j1pxbeasjdlvfsjqhzf6oqzeo1phtb]   Succeed to download the npm package @babel/plugin-proposal-logical-assignment-operators-7.13.8.tgz-7.13.8.
[DEBUG] [2021-07-06 08:41:50,258 +0000] - [CTX=sjgpwi107sx5to1j1pxbeasjdlvfsjqhzf6oqzeo1phtb]   resolved url in file = https://pkgs.dev.azure.com/_/_packaging/_/npm/registry/@babel/plugin-transform-parameters/-/plugin-transform-parameters-7.13.0.tgz
[DEBUG] [2021-07-06 08:41:50,258 +0000] - [CTX=sjgpwi107sx5to1j1pxbeasjdlvfsjqhzf6oqzeo1phtb]   resolved url in link = http://pkgs.dev.azure.com/@babel/plugin-transform-parameters/7.13.0
[DEBUG] [2021-07-06 08:41:51,633 +0000] - [CTX=sjgpwi107sx5to1j1pxbeasjdlvfsjqhzf6oqzeo1phtb]   npm.accessToken is not defined

We never changed the pipeline configuration

      - task: WhiteSource@21
        displayName: WhiteSource Bolt Scan
        inputs:
          cwd: '$(System.DefaultWorkingDirectory)'
          projectName: '$(projectName)'

Anyone noticed this too? What could we possibly do other than giving up on this plugin for another service?

Notice that the plugin has changed version from 21.3.2 to 21.6.2 since May 2021 without our action — usr-local-ΕΨΗΕΛΩΝ, Jul 06 '21 at 08:48
Same issue here. I noticed it does not happened with nuget packages as my api project builds fine. On front projects it takes from 45 minutes to an hour. I don't think it has to do with the version change in may since my pipelines have been working fine until yesterday. — Lionel D, Jul 06 '21 at 15:51

score 2 · Accepted Answer · answered Jul 08 '21 at 09:38

This is an official feedback from Whitesource support

Starting release 21.6.2, the WhiteSource scan is performed directly within the Azure DevOps pipeline. This means that the WhiteSource task is running a scan as part of your pipeline build.

Prior to this change, the WhiteSource task didn't perform the scan directly, it collected the relevant information, sent it to a remote WhiteSource server which was the one running the scan itself. Only when the scan on the remote server finished and the results were sent back, the WhiteSource risk report on Azure DevOps was displayed. This caused the WhiteSource report to be loaded after a long time and had several issues. Thus we decide to make the changes to a direct scan and this is a much more straightforward method for scanning, and the WhiteSource report is loaded much faster and many other improvements as well. However, it is important to understand that now that the scan is performed synchronously as part of the build (and not remotely asynchronously), the build time (and not the scan time) is increased compared to the previous releases.

So it looks like they pushed a major change without warning the users that the pipeline will take longer

Wahoo ! It doesn't take longer ! It takes ages as far as npm packages are involved. There is an Owasp dependency check plugin. I will give it a try. Other possible solutions: cache npm packages? Double check that dev dependencies are not part of the check? — Lionel D, Jul 08 '21 at 11:54
I'll speak to my boss about buying Snyk. I just deleted all WS pipeline tasks from all our projects — usr-local-ΕΨΗΕΛΩΝ, Jul 08 '21 at 11:55

Whitesource Bolt Azure DevOps plugin takes long time (and is too chatty)

1 Answers1