Nodejs: download a file into string via http, using async await syntax

Question

How do I download a file into memory via http in nodejs, without the use of third-party libraries?

This answer solves a similar question, but I don't need to write file to disk.

What type of file? https://stackoverflow.com/questions/19119040/how-do-i-save-and-restore-a-file-object-in-local-storage — Mellet, Feb 10 '21 at 18:52
Well, you can download a text file very easily with any library that does an http request such as `http.get()` built into nodejs. If it's not a text file and thus is encoded in some way, then you will have to be able to properly decode it. We would need to know more specifics about exactly what you're trying to download to help you there. For the best possible answer, provide us with the URL of the file you are trying to download so we can see exactly what it is. Also, why no third party libraries? — jfriend00, Feb 10 '21 at 19:00
@jfriend00 sometimes bundle size matters for nodejs, for example in lambdas. That's why built-in modules like `http` are preferable — Dan, Feb 10 '21 at 19:05
Bundling for AWS lamba would be a valid reason, though always trying to roll your own has its own hazards so obviously it's a tradeoff. — jfriend00, Feb 10 '21 at 19:11

jfriend00 · Accepted Answer · 2021-02-10T19:12:12.013

You can use the built-in http.get() and there's an example right in the nodejs http doc.

http.get('http://nodejs.org/dist/index.json', (res) => {
  const { statusCode } = res;
  const contentType = res.headers['content-type'];

  let error;
  // Any 2xx status code signals a successful response but
  // here we're only checking for 200.
  if (statusCode !== 200) {
    error = new Error('Request Failed.\n' +
                      `Status Code: ${statusCode}`);
  } else if (!/^application\/json/.test(contentType)) {
    error = new Error('Invalid content-type.\n' +
                      `Expected application/json but received ${contentType}`);
  }
  if (error) {
    console.error(error.message);
    // Consume response data to free up memory
    res.resume();
    return;
  }

  res.setEncoding('utf8');
  let rawData = '';
  res.on('data', (chunk) => { rawData += chunk; });
  res.on('end', () => {
    try {
      const parsedData = JSON.parse(rawData);
      console.log(parsedData);
    } catch (e) {
      console.error(e.message);
    }
  });
}).on('error', (e) => {
  console.error(`Got error: ${e.message}`);
});

This example assumes JSON content, but you can change the process in the end event handler to just treat the rawData as text and change the check for json contentType to whatever type you are expecting.

FYI, this is somewhat lower level code and is not something I would normally use. You can encapsulate it in your own function (perhaps with a promise wrapped around it) if you really don't want to use third party libraries, but most people use higher level libraries for this purpose that just make the coding simpler. I generally use got() for requests like this and there is a list of other libraries (all promise-based) here.

Nodejs: download a file into string via http, using async await syntax

1 Answers1