Node.js/NodeMailer/Express/Outlook smtp host - Concurrent connections limit exceeded

Question

Hope you are well. I am in the middle of working on an application that uses express and nodemailer. My application sends emails successfully, but the issue is, is that I cannot send the emails off one at a time in a manner that I'd like. I do not want to put an array of addresses into the 'to' field, I'd like each e-mail out individually.

I have succeeded in this, however there is an issue. It seems Microsoft has written some kind of limit that prevents applications from having more than a certain number of connections at a time. (link with explanation at end of document of post.)

I have tried to get around this by a number of expedients. Not all of which I'll trouble you with. The majority of them involve setInterval() and either map or forEach. I do not intend to send all that many e-mails - certainly not any to flirt with any kind of standard. I do not even want any HTML in my emails. Just plain text. When my application sends out 2/3 e-mails however, I encounter their error message (response code 432).

Below you will see my code.

• As you can see, I'm at the point where I've even been willing to try adding my incrementer into setInterval, as if changing the interval the e-mails fire at will actually help.
• Right now, this is sending out some e-mails, but I'm eventually encountering that block. This usually happens around 2/3 e-mails. It is strangely inconsistent however.

This is the first relevant section of my code.

  db.query(sqlEmailGetQuery, param)
    .then(result => {
      handleEmail(result, response);
    }).catch(error => {
      console.error(error);
      response.status(500).json({ error: 'an unexpected error occured.' });
    });
});

This is the second section of it.

function handleEmail(result, response) {
  const email = result.rows[0];
  let i = 0;

  email.json_agg.map(contact => {
    const msg = {
      from: process.env.EMAIL_USER,
      to: email.json_agg[i].email,
      subject: email.subject,
      text: email.emailBody + ' ' + i
    };
    i++;
    return new Promise((resolve, reject) => {
      setInterval(() => {
        transporter.sendMail(msg, function (error, info) {
          if (error) {
            return console.log(error);
          } else {
            response.status(200).json(msg);
            transporter.close();
          }
        });
      }, 5000 + i);
    });
  });
}

• I originally tried a simple for loop over the contacts iterable email.json_agg[i].email, but obviously as soon as I hit the connection limit this stopped working.
• I have come onto stackoverflow and reviewed questions that are similar in nature. For example, this question was close to being similar, but this guy has over 8000 connections and if you read the rule I posted by microsoft below, they implemented the connection rule after he made that post.
• I have tried setInterval with forEach and an await as it connects with each promise, but as this was not the source of the issue, this did not work either.
• I have tried similar code to what you see above, except I have set the interval to as long as 20 seconds.

As my understanding of the issue has grown, I can see that I either have to figure out a way to wait long enough so I can send another e-mail - without the connection timing out or break off the connections every time I send an e-mail, so that when I send the next e-mail I have a fresh connection. It seems to me that if the latter were possible though, everyone would be doing it and violating Microsofts policy.

Is there a way for me to get around this issue, and send 3 emails every say 3 seconds, and then wait, and send another three? The volume of e-mails is such that, I can wait ten seconds if necessary. Is there a different smtp host that is less restrictive?

Please let me know your thoughts. My transport config is below if that helps.

const transporter = nodemailer.createTransport({
  pool: true,
  host: 'smtp-mail.outlook.com',
  secureConnection: false,
  maxConnections: 1,
  port: 587,
  secure: false,
  tls: { ciphers: 'SSLv3' },
  auth: {
    user: process.env.EMAIL_USER,
    pass: process.env.EMAIL_PASS
  }
});

https://learn.microsoft.com/en-us/exchange/troubleshoot/send-emails/smtp-submission-improvements#new-throttling-limit-for-concurrent-connections-that-submitmessages

Answer 1

First off, the most efficient way to send the same email to lots of users is to send it to yourself and BCC all the recipients. This will let you send one email to the SMTP server and then it will distribute that email to all the recipients with no recipient being able to see the email address of any individual recipient.

Second, you cannot use timers to reliably control how many requests are running at once because timers are not connected to how long a given requests takes to complete so timers are just a guess at an average time for a request and they may work in some conditions and not work when things are responding slower. Instead, you have to actually use the completion of one request to know its OK to send the next.

If you still want to send separate emails and send your emails serially, one after the other to avoid having too many in process at a time, you can do something like this:

async function handleEmail(result) {
    const email = result.rows[0];
    for (let [i, contact] of email.json_agg.entries()) {
        const msg = {
          from: process.env.EMAIL_USER,
          to: contact.email,
          subject: email.subject,
          text: email.emailBody + ' ' + i
        };
        await transporter.sendMail(msg);
    }
}

If you don't pass transporter.sendMail() the callback, then it will return a promise that you can use directly - no need to wrap it in your own promise.

Note, this code does not send a response to your http request as that should be the responsibility of the calling code and your previous code was trying to send a response for each of the emails when you can only send one response and the previous code was not sending any response if there was an error.

This code relies on the returned promise back to the caller to communicate whether it was successful or encountered an error and the caller can then decide what to do with that situation.

You also probably shouldn't pass result to this function, but should instead just pass email since there's no reason for this code to know it has to reach into some database query result to get the value it needs. That should be the responsibility of the caller. Then, this function is much more generic.

If, instead of sending one email at a time, you want to instead send N emails at a time, you can use something like mapConcurrent() to do that. It iterates an array and keeps a max of N requests in flight at the same time.