Nodejs synchronous loops block execution

Question

when I try to run a function in background it blocks every other requests until it is done...

For example if I execute that function and then try to make a get request to a route that returns some information from the database then the response will come only after that function execution is done and I don't understand why.

This is the basic structure of my function that runs in background (it finds the 3rd party requests from a page and then look for the initiator request for each of them):

  const thirdPartyReq = [];
  let allRequests = [];

  const findInitiatorReq = async () => {
    allRequests = allRequests.reverse();
    for(const [_, request] of thirdPartyReq.entries()) {
      if(!request["Initiator Request"]) {
        const fullRequest = request['Request URL'];
        const parseUrl = new URL(fullRequest);
        let hostname = parseUrl.hostname || null;

        const domain = await extractDomain(hostname);
        let pathname = parseUrl.pathname || null;
        hostname = hostname.replace(/www./g, '')
        let checkUrl;

        const domainIndex = hostname.indexOf(domain) - 1;
        const subdomain = (hostname.substr(0, domainIndex));
        const queryString = parseUrl.search || '';
        const noProtocol = hostname + pathname + queryString;
        const noQueryString = hostname + pathname;
        const requestProcessing = [fullRequest, noProtocol, noQueryString, hostname];

        const requestIndex = allRequests.findIndex((el) => {
          return (el.url == request['Request URL'] && el.thirdParty);
        });

        for(const [_, query] of requestProcessing.entries()) {
          for(const [index, checkRequest] of allRequests.entries()) {
            if(index > requestIndex) {
              if(checkRequest.content && checkRequest.content.body) {
                const contentBody = checkRequest.content.body;
                if(contentBody.includes(query)) {
                  request['Initiator Request'] = checkRequest.url;
                }
              }
            }
          }
        }
      }
    }
  }

  for(const [pageIndex, page] of results.entries()) {
    const pageUrl = page.url;
    const requests = page.requests;
    const savedRequestUrls = [];
    let parseUrl = new URL(pageUrl);
    let hostname = parseUrl.hostname;
    let requestsCounter = 0;

    const pageDomain = await extractDomain(hostname);

    if(!urlList.includes(pageUrl)) {
      crawledUrls.push(pageUrl);
    }

    for(const [_, request] of Object.entries(requests)) {
      if(request.url.indexOf('data:') == -1) {
        parseUrl = new URL(request.url);
        hostname = parseUrl.hostname;
        let requestDomain = await extractDomain(hostname);

        const reqObj = await findThirdPartyReq(pageUrl, request, requestDomain);
        if(reqObj != null) {
          request.thirdParty = true;
          savedRequestUrls.push(reqObj);
        }

        // Store all requests that have a domain
        if(requestDomain) {
          request.page = pageUrl;
          allRequests.push(request);
          requestsCounter++;
        }
      }
    }

    findInitiatorReq();
  }

I noticed that everything will work well if I remove this part of code:

    for(const [_, query] of requestProcessing.entries()) {
      for(const [index, checkRequest] of allRequests.entries()) {
        if(index > requestIndex) {
          if(checkRequest.content && checkRequest.content.body) {
            const contentBody = checkRequest.content.body;
            if(contentBody.includes(query)) {
              request['Initiator Request'] = checkRequest.url;
            }
          }
        }
      }
    }

This is the route that calls the function:

router.get('/cookies',async (req, res) => {
   res.status(200).send(true);
   const cookies = await myFunc();
}

Can anyone please tell me why that function is blocking everything until it returns a response and how can I fix this?

Answer 1

Well, obviously you have a synchronous loop, which, of course, blocks execution. It will eventually block it anyway, as it has to perform several heavy operations. The response to the client is sent, but you still continue to work on some stuff, so other requests will have to wait.

A probable solution could be something like triggering another node process and handling stuff out there (something similar to a WebWorker in the browser)

You can try this library: async, there is a eachSeries method in it, meant specifically for handling big chunks of data/arrays. See the documentation for further information

Answer 2

The obvious answer here is to convert your function into an asynchronous one. There are already multiple answers here on StackOverflow about that topic. The gist: use asynchronous functions when you're elaborating some heavy task. Bear in mind that NodeJS is single threaded, thus the fact that sync functions block execution of other functions, is somewhat expected.

The tool you need to use to achieve asynchronous functions are: async/await (included without libraries/transpiling in the latest NodeJS LTS) and Promises. Forget about callbacks, since they are a really bad design.

How to use async / await in js:

• https://medium.com/@Abazhenov/using-async-await-in-express-with-node-8-b8af872c0016
• https://medium.freecodecamp.org/how-to-write-beautiful-node-js-apis-using-async-await-and-the-firebase-database-befdf3a5ffee

How to use Promises and what they are:

• Replacing callbacks with promises in Node.js
• understanding javascript promise object