Fastest way to make a million POST requests to a cloud function?

Question

I have an array with a length of one million. Each element is a string. I have a cloud function that takes a string and processes it. What is the fastest way to POST all million strings in my array to the cloud function? I don't care for the response of the cloud function. Ideally, it would POST, not wait for a response, then move on and POST the next one, iterating through the entire list as fast as possible. The issue is apparently with HTTP you cannot not wait for a response. You must wait for the response. Each cloud function takes about 10 seconds to execute. So if I need to wait for each response before moving to the next one, this would take 10 million seconds. Whereas if I could post each and not wait, I could probably run through the entire array in a few seconds.

`The issue is apparently with HTTP you cannot not wait for a response.` Not sure what you mean. Most libraries for doing http (eg, axios, fetch) will return a promise. You will only wait if you `await` that promise or call `.then` on it. Omit that, and your code will continue synchronously. There's a limit on how many http calls a browser will make at once, so maybe you're running into that, but i would expect that to only affect the network traffic, not your javascript. — Nicholas Tower, Mar 22 '22 at 01:07
You certainly don't have to wait for the response to request 1 before kicking off request 2. If you're have a specific problem, maybe you should show your code and explain what isn't working the way you expect, being clear on what environment your code is running under and the general constraints of your problem space. — Doug Stevenson, Mar 22 '22 at 01:07

jfriend00 · Accepted Answer · 2022-03-22T02:05:11.450

A lot of this has been covered before in prior questions/answers, but none that I found is a pure duplicate of what you're asking so I'll reference some that have come before and add some explanation. First the ones that have come before:

How to make millions of parallel http requests from nodejs app

How to fire off 1,000,000 requests

Is there a limit to how many promises can or should run concurrently when making requests

In Node js. How many simultaneous requests can I send with the "request" package

What is the limit of sending concurrent ajax requests with node.js?

How to loop many http requests with axios in node.js

Handling large number of outbound HTTP requests

Promise.all consumes all my RAM

Properly batch nested promises in Node

How can I handle a file of 30,000 urls without memory leaks?

First off, you can send a lot of parallel outbound requests. You do not have to wait for a prior response before sending the next one.

Second, you have resource limits on both client and server and ultimately, you will have to explore with testing your local configuration and your target server to find out where those resource limits are and then write your code to stay within those limits. There is no way to reliably send a request and then immediately kill the socket because you don't care about the response. If your socket gets queued by the target server (because you've already overwhelmed it), then killing the socket may drop it from the target server's queue before it gets processed by the target server.

Your local configuration will be limited by how many simultaneous sockets you can have open and how much memory you have (as each outbound request takes some amount of memory to keep track of).

The target server will be limited by its own resources. It may have protections built-in to limit how many posts/sec it can received from one particular source (rate limiting). It may have overall server protections against how many incoming requests at once it can handle. Typically servers protect themselves from overload by configuring things so that once an incoming request queue gets to a certain level, they just immediately hang up on new requests. The idea is to provide some level of protection of service and just deflect new requests when they come in too fast.

If this isn't your target server and there isn't any documentation about what its limits are supposed to be, then you will just have to test how many simutaneous requests you can have "in-flight" at the same time. If they implement rate limiting from a given source, then it's not uncommon that this might be a fairly low number such as 5. If no rate limiting, then you're really just trying to figure out what their http server can handle without causing it to drop connections in defense of service.

Once you figure out (with testing) how many simultaneous requests in flight the target server can comfortably handle, you will have to structure your code to deliver that. Usually, you would take an approach like is show in this mapConcurrent() function where you code things so that only N requests are in flight at the same time where N is a number you figured out experimentally by testing the target server.

Relevant pieces of helper code:

mapConcurrent(array, maxConcurrent, fn)

rateLimitMap(array, requestsPerSec, maxInFlight, fn)

runN(fn, limit, cnt, options)

pMap(array, fn, limit)

And, if you want a pre-made library, the async library contains a bunch of control flow helpers like these.