Synchronous Requests in Node.js

Question

How could I make the 'request' module in Node.js load things in a synchronous fashion? The best advice I've seen is to somehow use the callback to get the function to not return itself until it is done. I am trying to use the 'request' function inline in code (things need to be processed based on that data that can't be placed in callbacks).

So how could I use the callback of the 'request' module to keep it from returning itself until it is finished with loading the resource?

What I'm doing is running a loop that downloads two values from an API, and then has to do some math based on those values. While the math could be done in callbacks... the loop would advance without the values it needs to perform the next operation. (So stopping the loop from advancing until the data is ready would solve the issue)

    /* loop */ {
         /* URL Generation */


    request( {url: base + u_ext}, function( err, res, body ) {
        var split1 = body.split("\n");
        var split2 = split1[1].split(", ");
        ucomp = split2[1];
    });

    request( {url: base + v_ext}, function( err, res, body ) {
        var split1 = body.split("\n");
        var split2 = split1[1].split(", ");
        vcomp = split2[1];
    });

    /* math which needs to be after functions get variables and before loop advances */
    }

Answer 1

The short answer is: don't. (...) You really can't. And that's a good thing

I'd like to set the record straight regarding this:

NodeJS does support Synchronous Requests. It wasn't designed to support them out of the box, but there are a few workarounds if you are keen enough, here is an example:

var request = require('sync-request'),
    res1, res2, ucomp, vcomp;

try {
    res1 = request('GET', base + u_ext);
    res2 = request('GET', base + v_ext);
    ucomp = res1.split('\n')[1].split(', ')[1];
    vcomp = res2.split('\n')[1].split(', ')[1];
    doSomething(ucomp, vcomp);

} catch (e) {}

When you pop the hood open on the 'sync-request' library you can see that this runs a synchronous child process in the background. And as is explained in the sync-request README it should be used very judiciously. This approach locks the main thread, and that is bad for performance.

However, in some cases there is little or no advantage to be gained by writing an asynchronous solution (compared to the certain harm you are doing by writing code that is harder to read).

This is the default assumption held by many of the HTTP request libraries in other languages (Python, Java, C# etc), and that philosophy can also be carried to JavaScript. A language is a tool for solving problems after all, and sometimes you may not want to use callbacks if the benefits outweigh the disadvantages.

For JavaScript purists this may rankle of heresy, but I'm a pragmatist so I can clearly see that the simplicity of using synchronous requests helps if you find yourself in some of the following scenarios:

1. Test Automation (tests are usually synchronous by nature).

2. Quick API mash-ups (ie hackathon, proof of concept works etc).

3. Simple examples to help beginners (before and after).

Be warned that the code above should not be used for production. If you are going to run a proper API then use callbacks, use promises, use async/await, or whatever, but avoid synchronous code unless you want to incur a significant cost for wasted CPU time on your server.

Answer 2

In 2018, you can program the "usual" style using async and await in Node.js.

Below is an example, that wraps request callback in a promise and then uses await to get the resolved value.

const request = require('request');

// wrap a request in an promise
function downloadPage(url) {
    return new Promise((resolve, reject) => {
        request(url, (error, response, body) => {
            if (error) reject(error);
            if (response.statusCode != 200) {
                reject('Invalid status code <' + response.statusCode + '>');
            }
            resolve(body);
        });
    });
}

// now to program the "usual" way
// all you need to do is use async functions and await
// for functions returning promises
async function myBackEndLogic() {
    try {
        const html = await downloadPage('https://microsoft.com')
        console.log('SHOULD WORK:');
        console.log(html);

        // try downloading an invalid url
        await downloadPage('http://      .com')
    } catch (error) {
        console.error('ERROR:');
        console.error(error);
    }
}

// run your async function
myBackEndLogic();

Answer 3

Though asynchronous style may be the nature of node.js and generally you should not do this, there are some times you want to do this.

I'm writing a handy script to check an API and want not to mess it up with callbacks.

Javascript cannot execute synchronous requests, but C libraries can.

https://github.com/dhruvbird/http-sync

Answer 4

Aredridels answer is relatively good (upvoted it), but I think it lacks the loop equivalent. This should help you:

Sync code equivalent:

while (condition) {
  var data = request(url);
  <math here>
}
return result;

Async code for serial execution:

function continueLoop() {
  if (!condition) return cb(result);
  request(url, function(err, res, body) {
    <math here>
    continueLoop()
  })
}
continueLoop()

Answer 5

You should take a look at library called async

and try to use async.series call for your problem.

Answer 6

See sync-request: https://github.com/ForbesLindesay/sync-request

Example:

var request = require('sync-request');
var res = request('GET', 'http://example.com');
console.log(res.getBody());

Answer 7

These days if you want to do things sequentially, you'd do something like:

for (const task of tasks) {
  const response = await request(...);
}

This code (above) runs the requests in order, but is not synchronous (meaning it does not block the main thread). If you really need synchronous, then most of these libraries boil down to something like:

const child = require('child_process');
const buffer = child.execSync(`curl https://example.com/foo.json`);
const data = JSON.parse(buffer.toString('utf8'));

Answer 8

You can use retus to make cross-platform synchronous HTTP requests. It's a library based on sync-request but with a few comfort features I added:

const retus = require("retus");

const { body } = retus("https://google.com");
//=> "<!doctype html>..."

That's it!

Answer 9

You can do something exactly similar with the request library, but this is sync using const https = require('https'); or const http = require('http');, which should come with node.

Here is an example,

const https = require('https');

const http_get1 = {
    host : 'www.googleapis.com',
    port : '443',
    path : '/youtube/v3/search?arg=1',
    method : 'GET',
    headers : {
        'Content-Type' : 'application/json'
    }
};

const http_get2 = {
host : 'www.googleapis.com',
    port : '443',
    path : '/youtube/v3/search?arg=2',
    method : 'GET',
    headers : {
        'Content-Type' : 'application/json'
    }
};

let data1 = '';
let data2 = '';

function master() {

    if(!data1)
        return;

    if(!data2)
        return;

    console.log(data1);
    console.log(data2);

}

const req1 = https.request(http_get1, (res) => {
    console.log(res.headers);

    res.on('data', (chunk) => {
        data1 += chunk;
    });

    res.on('end', () => {
        console.log('done');
        master();
    });
});


const req2 = https.request(http_get2, (res) => {
    console.log(res.headers);

    res.on('data', (chunk) => {
        data2 += chunk;
    });

    res.on('end', () => {
        console.log('done');
        master();
    });
});

req1.end();
req2.end();

Answer 10

The easiest solution I came up for myself was to use the node's native "child_request" and simply call exec with a simple curl command inside.... all dependencies and asynchronity hassle me way too much for the simple rare cases in which I simply want to "save the http response to a variable in node"

Answer 11

As of this writing all answers are either:

1. Sync but with a control flow (such as using async library)
2. Sync, but blocking [which means that all other threads on Node are stopped, which is bad performance wise], such as retus or sync-request.
3. Out of date, such as http-request.

Check out the non-blocking sync-request, based on deasync.

Or see some of the answers in this thread, such as this, which use deasync directly.

Answer 12

The short answer is: don't. If you want code that reads linearly, use a library like seq. But just don't expect synchronous. You really can't. And that's a good thing.

There's little or nothing that can't be put in a callback. If they depend on common variables, create a closure to contain them. What's the actual task at hand?

You'd want to have a counter, and only call the callback when the data is there:

var waiting = 2;
request( {url: base + u_ext}, function( err, res, body ) {
    var split1 = body.split("\n");
    var split2 = split1[1].split(", ");
    ucomp = split2[1];
    if(--waiting == 0) callback();
});

request( {url: base + v_ext}, function( err, res, body ) {
    var split1 = body.split("\n");
    var split2 = split1[1].split(", ");
    vcomp = split2[1];
    if(--waiting == 0) callback();
});

function callback() {
    // do math here.
}

Update 2018: node.js supports async/await keywords in recent editions, and with libraries that represent asynchronous processes as promises, you can await them. You get linear, sequential flow through your program, and other work can progress while you await. It's pretty well built and worth a try.