Socket hangup due to wrong handling of promises

Question

I have script to move data from one platform to another. The source db allows only 100 records to be fetched in a single request. So I created a routine to fetch by batches of 100 which works fine I guess.

Now I try to process each records of 100 and do the necessary transformations (which involves axios call to get certain data) and create a record in firebase firestore. Now when I run this migration in firebase express node, I get socket hang up ECONNRESET.

I know this is caused by wrong handling of promises.

Here is what my code looks like:

import { scrollByBatches } from "../helpers/migrations/apiScroll";
import { createServiceLocation } from "../helpers/locations";

const mapServiceLocationData = async (serviceLocation: any, env: string) => {
  try {
    const migratedServiceLocation: any = {
      isMigrated: true,
      id: serviceLocation._id,
    };


    if (serviceLocation.list?.length) {
      await Promise.all(serviceLocation.ids.map(async (id: string) => {
        const { data } = await dbEndPoint.priceMultiplier({ id });  // error says socket hangup on this call
        let multiplierUnit;
        let serviceType;
        if (data.response._id) {
          multiplierUnit = data.response;
          const result = await dbEndPoint.serviceType({ id: multiplierUnit.service_custom_service_type }); // error says socket hangup on this call
          if (result.data.response._id) {
            serviceType = result.data.response.type_text;
            migratedServiceLocation.logs = [...multiplierUnit.history_list_text, ...migratedServiceLocation.logs];
          }
        }
      }));
    }

    await createServiceLocation(migratedServiceLocation); // create record in destination db
  } catch (error) {
    console.log("Error serviceLocation: ", serviceLocation._id, JSON.stringify(error));
  }
  return null; // is this even necessary?
};

export const up = async () => {
  try {
    // get 100 docs from source db => process it.. => fetch next 100 => so on...
    await scrollByBatches(dbEndPoint.serviceLocation, async (serviceLocations: any) => {
      await Promise.all(
          serviceLocations.map(async (serviceLocation: any) => {
            await mapServiceLocationData(serviceLocation);
          })
      );
    }, 100);
  } catch (error) {
    console.log("Error", JSON.stringify(error));
  }
  return null; // is this even necessary?
};

The error I get in firebase functions console is:

For clarity on how the fetch by batches looks like:

const iterateInBatches = async (endPoint: any, limit: number, cursor: number, callback: any, resolve: any, reject: any) => {
  try {
    const result = await endPoint({ limit, cursor });
    const { results, remaining }: any = result.data.response;

    if (remaining >= 0) {
      await callback(results);
    }
    if ((remaining)) {
      setTimeout(() => {
        iterateInBatches(endPoint, limit, (cursor + limit), callback, resolve, reject);
      }, 1000); // wait a second
    } else {
      resolve();
    }
  } catch (err) {
    reject(err);
  }
};

export const scrollByBatches = async (endPoint: any, callback: any, limit: number, cursor: number = 0) => {
  return new Promise((resolve, reject) => {
    iterateInBatches(endPoint, limit, cursor, callback, resolve, reject);
  });
};

What am I doing wrong? I have added comments in the code sections for readability.

Thanks.

Answer 1

There are two cases when socket hang up gets thrown:

When you are a client

When you, as a client, send a request to a remote server, and receive no timely response. Your socket is ended which throws this error. You should catch this error and decide how to handle it: whether to retry the request, queue it for later, etc.

When you are a server/proxy

When you, as a server, perhaps a proxy server, receive a request from a client, then start acting upon it (or relay the request to the upstream server), and before you have prepared the response, the client decides to cancel/abort the request.

I would suggest a number of possibilities for you to try and test that might help you solve your issue of ECONNRESET :

• If you have access to the source database, you could try looking there for some logs or metrics. Perhaps you are overloading the service.

• Quick and dirty solution for development: Use longjohn, you get long stack traces that will contain the async operations. Clean and correct solution: Technically, in node, whenever you emit an 'error' event and no one listens to it, it will throw the error. To make it not throw, put a listener on it and handle it yourself. That way you can log the error with more information.

• You can also set NODE_DEBUG=net or use strace. They both provide you what the node is doing internally.

• You could restart your server and run the connection again, maybe your server crashed or refused the connection most likely blocked by the User Agent.

• You could also try running this code locally, instead of in cloud functions to see if there is a different result. It's possible that the RSG/google network is interfering somehow.

• You can also have a look at this GitHub issue and stackoverflow thread to see the common fixes for the ECONNRESET issue and see if those help resolve the issue.