We have a specific piece of NodeJS software which downloads files to it's local filesystem. The files can very from a few KB's to GB's. Sometimes, for no apparent reason the download speeds becomes really slow. The actual connection is quite decent and stable. The thing curious thing is, the slow download will only happen at the very beginning of a download and will not scale back up to a decent speed.
This problem happens intermittently, it can happen from the first download or after another one has finished. The code downloads files sequentially, so no multiple downloads at the same time. When the problem occurs, a simple stop of the NodeJS process and restart resolves it.
Below an output sample:
content_VIDEO_a67a8535-2db4-4d36-9869-f16b1916df77_20180717_en-US.zip
[ ] 0%
0.00 of 3.26 GB
------------------------------------
content_VIDEO_a67a8535-2db4-4d36-9869-f16b1916df77_20180717_en-US.zip
[================================== ] 34%
1.13 of 3.26 GB
------------------------------------
content_VIDEO_a67a8535-2db4-4d36-9869-f16b1916df77_20180717_en-US.zip
[======================================================================= ] 71%
2.33 of 3.26 GB
content_VIDEO_a67a8535-2db4-4d36-9869-f16b1916df77_20180717_en-US.zip
[================================================================================================= ] 97%
3.17 of 3.26 GB
------------------------------------
Download of content_VIDEO_a67a8535-2db4-4d36-9869-f16b1916df77_20180717_en-US.zip completed.
Downloading next file
content_VIDEO_51bd27ee-12c5-49e0-b321-f0be58d647b3_20180717_en-US.zip
[ ] 0%
0.00 of 3.48 GB
------------------------------------
content_VIDEO_51bd27ee-12c5-49e0-b321-f0be58d647b3_20180717_en-US.zip
[ ] 0%
0.00 of 3.48 GB
------------------------------------
content_VIDEO_51bd27ee-12c5-49e0-b321-f0be58d647b3_20180717_en-US.zip
[ ] 0%
0.00 of 3.48 GB
------------------------------------
content_VIDEO_51bd27ee-12c5-49e0-b321-f0be58d647b3_20180717_en-US.zip
[ ] 0%
0.00 of 3.48 GB
------------------------------------
content_VIDEO_51bd27ee-12c5-49e0-b321-f0be58d647b3_20180717_en-US.zip
[ ] 0%
0.00 of 3.48 GB
------------------------------------
content_VIDEO_51bd27ee-12c5-49e0-b321-f0be58d647b3_20180717_en-US.zip
[ ] 0%
0.00 of 3.48 GB
------------------------------------
content_VIDEO_51bd27ee-12c5-49e0-b321-f0be58d647b3_20180717_en-US.zip
[ ] 0%
0.01 of 3.48 GB
As you can see in the above output, the first (or previous file) downloads with normal speed. The second one, which is on the same server is almost not downloading at all.
For the download the library Request(v2.87.0) is used along with Request-progress (v3.0.0) to show the download progress. The issue happens on both Windows as well as Linux platforms. We've noticed to issue on multiple NodeJS versions too, such as 8.11.1 & 8.11.3
The code used to download:
function download (url, pathToFile) {
const context = this;
return new Promise((resolve, reject) => {
const options = {
// Set 10 minutes timeout for connect / fetch
timeout: 600000,
headers: {}
};
const req = request.get(url, options);
progress(req, {
throttle: 60000, // Throttle the progress event, defaults to 1000ms
delay: 0 // Only start to emit after a delay of some milliseconds - default is 0ms
})
.on('progress', (state) => {
showProgress(state);
})
.on('error', (error) => {
req.abort();
reject(error);
})
.on('response', (response) => {
response.on('end', resolve);
})
.pipe(fs.createWriteStream(pathToFile, { flags: options.headers.Range ? 'a' : 'w' }));
});
};
function showProgress(state) {
let progressIndicator = '';
let balanceIndicator = '';
let progress = 0;
const percent = parseInt(state.percent * 100);
for (progress = 1; progress <= percent; progress++) {
progressIndicator += '=';
}
for (let balance = progress; balance <= 100; balance++) {
balanceIndicator += ' ';
}
let totalSize = ((state.size.total / 1024) / 1024).toFixed(2);
let receivedSize = ((state.size.transferred / 1024) / 1024).toFixed(2);
let sizeUnit = 'MB';
if (totalSize >= 1024) {
totalSize = (totalSize / 1024).toFixed(2);
receivedSize = (receivedSize / 1024).toFixed(2);
sizeUnit = 'GB';
}
console.log(`[${progressIndicator}${balanceIndicator}]`, `${percent}%`);
console.log(receivedSize, 'of', totalSize, sizeUnit, '\n');
console.log('------------------------------------');
console.log(' ');
}
There is no error, it's just downloading really slow, so it's very hard to debug where this issue is coming from.
Any suggestions are welcome
