Mock testing a function that downloads a file using node-fetch?

Question

This answer has the following way to download a file using the node-fetch library https://stackoverflow.com/a/51302466:

const downloadFile = (async (url, path) => {
  const res = await fetch(url);
  const fileStream = fs.createWriteStream(path);
  await new Promise((resolve, reject) => {
      res.body.pipe(fileStream);
      res.body.on("error", reject);
      fileStream.on("finish", resolve);
    });
});

How to mock this if a function has this method as part of it? I'm running into weird problems:

unit.test.js

import { PassThrough } from 'stream';
import { createWriteStream, WriteStream } from 'fs';
import fetch, { Response } from 'node-fetch';

import mocked = jest.mocked;


jest.mock('node-fetch');
jest.mock('fs');

describe('downloadfile', () => {
  const mockedFetch = fetch as jest.MockedFunction<typeof fetch>;
  const response = Promise.resolve({
    ok: true,
    status: 200,
    body: {
      pipe: jest.fn(),
      on: jest.fn(),
    },
  });

  const mockWriteable = new PassThrough();
  mocked(createWriteStream).mockReturnValueOnce(mockWriteable as unknown as WriteStream);

  mockedFetch.mockImplementation(() => response as unknown as Promise<Response>);
  it('should work', async () => {
      await downloadfile();
    }),
  );
});

Throws:

Cannot read property 'length' of undefined
TypeError: Cannot read property 'length' of undefined
    at GlobSync.Object.<anonymous>.GlobSync._readdirEntries (//node_modules/glob/sync.js:300:33)
    at GlobSync.Object.<anonymous>.GlobSync._readdir (//node_modules/glob/sync.js:288:17)
    at GlobSync.Object.<anonymous>.GlobSync._processReaddir (//node_modules/glob/sync.js:137:22)
    at GlobSync.Object.<anonymous>.GlobSync._process (/api/node_modules/glob/sync.js:132:10)

What could the solutions be?

Answer 1

I see that the error is not related to tests. It is more about your test runner. Glob is a library to search files in your project. Check your test run command. Try to escape quotes \ double quotes.

Your package JSON will give more information that your test files.

There are some people that report this problem can be fixed by reinstalling node modules Try to

1. Remove node_modules folder
2. Remove the file package-lock.json
3. Run npm install to re-install package dependencies.

Answer 2

Your base function has a fetch function which expects a URL from which the response body(data) is piped to the write stream and then written to the file system.

Mocking fetch is a bit tedious as jest mock has limited support for ES modules, More info here and here. So better test it by letting fetch download a file from a URL and then test the integrity of the downloaded file.

Now, calling the function inside your test case as follows:

const fileURL = "https://interactive-examples.mdn.mozilla.net/media/examples/lizard.png";
const fileDownloadPath = "/home/<your-username>/Downloads/downloaded_lizard.png";

// ----> Executing the function to be tested.
await downloadFile(fileURL, fileDownloadPath);

Next, after downloading the file we can verify if the file exists at the download path and then test for it's integrity.

Integrity checking involves that the downloaded file checksum (md5 in our case) should match the checksum retrieved from the server.

If you are using a AWS S3 file URL then it should most probably return the checksum in the header named "ETag", also note that this is not true in every case but for small files at least this holds true.

If checksum is not available from the URL you can download the file using the specified URL by yourself and calculate its md5 checksum on the local machine then assign this checksum to a const variable inside the test case which can then be further used to test the integrity of the file.

const access_res = await fsp.access(fileDownloadPath, fs.constants.F_OK); // File should be accessible
expect(access_res).toEqual(undefined); // denotes success

// Compare md5 checksum of downloaded file with server file.
const file_buffer = fs.readFileSync(fileDownloadPath);
const downloaded_file_md5_checksum = createHash('md5').update(file_buffer).digest('hex');
const response_obj = await fetch(fileURL, { method: 'HEAD' });
const file_etag = JSON.parse(response_obj.headers.get('ETag'));
expect(downloaded_file_md5_checksum).toEqual(file_etag);  // Should match

Here is the full working example using ES modules:

import { jest } from '@jest/globals';
import fs from 'node:fs';
import fsp from 'node:fs/promises';
const { createHash } = await import('node:crypto');
import fetch from 'node-fetch';
import { downloadFile } from '../index';

// jest.setTimeout(15000);  // in ms. Override if downloading big files.

describe("Download service", () => {
  it("should download the file from provided url", async () => {

    const fileURL = "https://interactive-examples.mdn.mozilla.net/media/examples/lizard.png";
    const fileDownloadPath = "/home/<your-username>/Downloads/downloaded_lizard.png";

    // ----> Executing the function to be tested.
    await downloadFile(fileURL, fileDownloadPath);

    // ----> Testing if the file exists on fs.
    const access_res = await fsp.access(fileDownloadPath, fs.constants.F_OK);
    expect(access_res).toEqual(undefined); // denotes success

    // ----> Comparing downloaded file checksum with server file checksum.
    const file_buffer = fs.readFileSync(fileDownloadPath);
    const downloaded_file_md5_checksum = createHash('md5').update(file_buffer).digest('hex');
    const response_obj = await fetch(fileURL, { method: 'HEAD' });  // Fetching checksum from server.
    const file_etag = JSON.parse(response_obj.headers.get('ETag'));
    expect(downloaded_file_md5_checksum).toEqual(file_etag);  // Should match

    // ----> Finally remove downloaded file from local system.
    const rm_res = await fsp.unlink(fileDownloadPath);
    expect(rm_res).toEqual(undefined); // denotes success
  })
})