-4

I'm using node.js 8.4.0 and I want to write a function that performs a HTTP request and returns the result from the server. This means the function should wait until the request is complete.

I'm new to Javascript, but I've read other answers about Javascript being asynchronous and about how I should change my mental model and start using callbacks. Please, don't bother restating that, because it's childish and myopic. My need is fully legitimate and also a very simple one. If there's no way I can write the program the way I want, I'll ditch the whole thing and use a different language, the only problem being a particular library which exists only for JS.

As much as possible, I'm looking for a simple, general, robust and portable way to wait for a callback. Ideally not just for http, but also for any other async stuff. I know asynchronous programming is a great thing, but only when you really need it. I don't.

EDIT: see below.

jurez
  • 4,436
  • 2
  • 12
  • 20
  • 5
    No. no no no. NO. `How to hack a single-threaded asynchronous engine and force it to perform synchronous stuff` = big red alert to me. Instead, embrace the power of asynchronism and learn how to deal with it. Or go back to PHP. – Jeremy Thille Sep 06 '17 at 07:31
  • I'd bet you also *do* need it, but you're not willing to get used to it yet ;-) Just my two cents. – Rob Sep 06 '17 at 07:33
  • 3
    `I'm looking for a way to wait for a callback` --> You don't _wait_ for a callback. A callback is a function that gets triggered whenever the stuff you asked for is done. In the meantime, the script execution is not blocked. That's the very principle of asynchronism. – Jeremy Thille Sep 06 '17 at 07:35
  • 2
    If you google synchronous HTTP requests this is what you get first: https://www.npmjs.com/package/sync-request – Ray Toal Sep 06 '17 at 07:35
  • JavaScript isn't asynchronous. But many of the things we *use* it for are. Big difference. Node, in particular, uses only a single JavaScript thread and so using asynchronous handling of long-running tasks (like I/O) is very important. – T.J. Crowder Sep 06 '17 at 07:35
  • 7
    *"Please, don't bother restating that, because it's childish and myopic."* Well, that's a great way to inspire people to help you. – T.J. Crowder Sep 06 '17 at 07:36
  • 2
    Ditch the whole thing and use another language. I'm serious. Node.js and javascript is my go-to tool but you MUST change your mental model and learn async programming. If you refuse to do this then js is not for you. You can't really change how event-oriented single-threaded systems work. So you must change how programmers think about code. – slebetman Sep 06 '17 at 07:45
  • Thanks for you input. I wasn't going to discourage anybody, only trying to avoid useless answers. I've done quite a bit of multithreading myself in other languages, including implementing workers, synchronization mechanisms, thread pools etc. I realize each thing has its pros and cons, and I've googled it and asked around to no avail. But this is really turning into a religious war instead of helping. Sad. – jurez Sep 06 '17 at 07:45
  • 2
    @jurez It's not a religious war. Node.js is single-threaded. This is a technical limitation. Either accept the technical limitation or use another language. Doing multithreaded programming is not a good preparation for node.js. Doing non-blocking I/O / asynchronous programming / concurrent programming (they are all different names for the same thing) in frameworks like Python's Twisted or Java's Playframework are good preparation for node.js – slebetman Sep 06 '17 at 07:48
  • OK, if it's single-threaded, how come the function continues to execute while HTTP request is being made at the same time? What am I missing here? – jurez Sep 06 '17 at 07:54
  • 2
    @jurez: If you would describe your problem we could take a look at it. You *can* do everything with async JS that you can do with synchronous code. I'm absolutely sure that there's a good async way to resolve your issue. – Rob Sep 06 '17 at 07:54
  • I'm not refusing async programming, I'm just saying it makes things unneccessarily unreadable and complicated in my case where the nature of the problem is synchronous. I'm puzzled by the fact that there is no such thing as promise.waitFor() or something. Or at least I couldn't find it. – jurez Sep 06 '17 at 07:57
  • 1
    @jurez: For a low level explanation of how async work without threads see: https://stackoverflow.com/questions/29883525/i-know-that-callback-function-runs-asynchronously-but-why, for a high level explanation see: https://stackoverflow.com/questions/30204750/nodejs-asychronous-i-o-execution – slebetman Sep 06 '17 at 08:02
  • 1
    *"But this is really turning into a religious war instead of helping."* Well, you reap what you sow. (And I'm seeing quite a lot of help above, actually.) – T.J. Crowder Sep 06 '17 at 08:10
  • @jurez Also note that you will have exactly this same problem if you decide to go asynchronous in other languages like Java or Python or C++. For example, with Java you have to deal with CompletionStage (Futures) and you can't wait for things to complete but have to pass a lambda (or Runnable of Function) to the async function instead. – slebetman Sep 06 '17 at 08:10
  • slebetman's comment about Node being single-threaded is *mostly* true, in that it uses a single JavaScript thread. It also uses async I/O behind the scenes, and responds to async I/O callbacks, which is how I/O can continue even with only one thread. (I think it may well have other non-JavaScript threads, but we don't care at the JavaScript level.) If you want to write sync-seeming code in NodeJS, use promises and [`async`/`await` syntax](https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Statements/async_function), which recent versions of Node support. – T.J. Crowder Sep 06 '17 at 08:12
  • `there is no such thing as promise.waitFor() or something` --> Uh, and what about `promise.then()` ? – Jeremy Thille Sep 06 '17 at 08:20
  • @JeremyThille: Fairly sure he/she means a *synchronous* wait. – T.J. Crowder Sep 06 '17 at 08:26
  • No, I believe they were first looking for a way to resolve it asynchronously, like `promise.waitFor() or something`, but couldn't find it, so they resolved to do it synchronously, hence the original question. – Jeremy Thille Sep 06 '17 at 08:29
  • With promise.then(), you still have only a promise. You cannot make a function "readHttpFromServer" which would return the actual result after it has ben read from the server. What I was hoping to do is the equivalent of "thread join" or "wait for process". So the thread executing the function would sleep and got notified by the thread performing the callback. As the underlying CPU still uses threads and contexts, I cannot see why this could not be possible with asynchronous model. But as a JS beginner, I realise I might be missing something..If this is not possible, please explain why. – jurez Sep 06 '17 at 08:45
  • `You cannot make a function "readHttpFromServer" which would return the actual result after it has ben read from the server` --> So... what would be the point of any server-side code then? Of course you can. `request("http://someapi.com").then(result => console.log(result))` Tadaaaaa, an asynchronous call, and, cherry on the cake, you even have access to the _result_ of this call! Increbible, isn't it? :) – Jeremy Thille Sep 06 '17 at 09:02

2 Answers2

2

If you feel you must issue a synchronous HTTP request...

...as Ray Toal pointed out, there's an npm package that does that for you by offloading it to a child process and using spawnSync to synchronously wait for that process to complete.

It has negative implications, which is why the advice is not to use synchronous I/O requests in Node, even though the Node API provides a few (fileReadSync, etc.). Node uses a single JavaScript thread. By blocking that thread on a long-running process (an HTTP request), you prevent that thread from doing anything else while the request is being processed.


If you just want to write synchronous-looking code...

...I'd suggest using promises and async/await syntax instead. async/await syntax brings asynchronous processing to code that looks synchronous, providing the benefits of asynchronous processing with synchronous-like syntax. Recent versions of Node use recent versions of the V8 JavaScript engine, which supports async/await.

Node predates Promises, and uses its own API conventions for callbacks, but there are packages such as promisify that can convert Node-callback style APIs to Promise-based ones. promisify works at the API level, so with a couple of lines of code you can convert the entire fs module's API for instance. As time moves forward, we can expect promises to be built-in to new packages (and I suspect retrofitted to the standard Node API as well).

For http.request, it's a bit more complicated than just updating the API as there are events to respond to. But "a bit more" is all it is. Here's a quick-and-simple version which also adds automatic handling of the Content-Length header:

const requestPromise = (options, postData = null) => new Promise((resolve, reject) => {
  const isPostWithData = options && options.method === "POST" && postData !== null;
  if (isPostWithData && (!options.headers || !options.headers["Content-Length"])) {
    // Convenience: Add Content-Length header
    options = Object.assign({}, options, {
      headers: Object.assign({}, options.headers, {
        "Content-Length": Buffer.byteLength(postData)
      })
    });
  }
  const body = [];
  const req = http.request(options, res => {
    res.on('data', chunk => {
      body.push(chunk);
    });
    res.on('end', () => {
      res.body = Buffer.concat(body);
      resolve(res);
    });
  });

  req.on('error', e => {
    reject(e);
  });

  if (isPostWithData) {
    req.write(postData);
  }
  req.end();
});

With that in our toolkit, we can make an asynchronous request within an async function like this:

try {
    const res = await requestPromise(/*...options...*/, /*...data if needed...*/);
    console.log(res.body.toString("utf8"));
    // ...continue with logic...
} catch (e) {
    console.error(e);
}

As you can see, the logical flow of the code is as it would be for synchronous code. But the code is not synchronous. The JavaScript thread runs the code up to and including the expression that's the operand of the first await, then does other things while the asynchronous process runs; later when the asynchronous process is complete, it picks up where it left off, assigning the result to res and then executing the console.log, etc. — up until the next await (if any). If the result of the promise await consumes is a rejection, it's processed as an exception and passed to the catch in the try/catch.

Here's the example from http.request using our requestPromise above:

try {
  const res = await requestPromise(
    {
      hostname: 'www.google.com',
      port: 80,
      path: '/upload',
      method: 'POST',
      headers: {
        'Content-Type': 'application/x-www-form-urlencoded'
      }
    },
    querystring.stringify({
      'msg': 'Hello World!'
    }
  );
  console.log(res.body.toString("utf8"));
} catch (e) {
  console.error(e);
}

To use await, you have to be in an async function. You'd probably just wrap your entire module code in one:

(async () => {
  // ...module code here...
})();

If there's any possibility of your code not catching an error, add a catch-all catch handler at the end:

(async () => {
  // ...module code here...
})().catch(e => { /* ...handle the error here...*/ });
T.J. Crowder
  • 1,031,962
  • 187
  • 1,923
  • 1,875
0

For the reference and for anybody stumbling at this in the future, this is the explanation I was looking for:

There is no such thing as synchronous requests/join/waitFor in Javascript because everything runs on the same thread, even callbacks. Callbacks are added to event queue and processed after thread returns from outer scope. Waiting for whatever, had it been possible, would essentially cause a deadlock.

This video explains it nicely: Philip Roberts: What the heck is the event loop anyway?

Thanks guys and keep up the promises!

jurez
  • 4,436
  • 2
  • 12
  • 20