Out of Memory sending mass emails using SmtpClient

Question

We have been having issues when sending mass emails out (roughly 60k ~ 70k). We usually get the Out of Memory exception at the 30k mark. We loop through a list of clients and we modify a string by passing parameters according to the client which is later loaded into the body of the MailMessage object. This error happens when we use threads to accelerate the process. We dont get the error when not using threads however the process takes about 9 hours, which is not acceptable to us. What could we do to control memory behaviour or anything that could get through the whole list without having this error? Please see parts of our code below:

foreach (var candidate in z)
        {
            var mailer1 = new MailerProto();
            string jobList = "";
            foreach (var item in candidate)
            {
                string title = item.title;
                jobList += "<a href='" + "https://mydomain/" + "job/" + title + "-" + item.jobId + "?emailId=" + item.accountId + "'>" + title + " </a><br class='br' />";
            }
            string[] args = new string[]
            {          
                AppConfig.Url,
                candidate.FirstOrDefault().firstName,
                jobList,
                candidate.FirstOrDefault().accountId.ToString()
            };

            tasks[taskCounter] = mailer1.SendEmail(
                                                  from:     AppConfig.NoReplyMail,
                                                  to:       candidate.FirstOrDefault().email,
                                                  subject:  "We have found jobs just for you!",
                                                  htmlContent: content,
                                                  args:     args,
                                                  skipSubscription: true
                                                 );
            taskCounter += 1;
            if (taskCounter == 5000)
            {
                Task.WhenAll(tasks);
                tasks = new Task[5000];
                taskCounter = 0;
                GC.Collect();
                //System.Threading.Thread.Sleep(500);
            }
        }

Then part of the SendMail function

for (int i = 0; i < args.Length; i++)
        {
            htmlContent = htmlContent.Replace("{" + i + "}", args[i]);
        }

        try
        {
            using (SmtpClient smtpClient = new SmtpClient())
            {

                MailMessage messageMail = new MailMessage();
                messageMail.To.Add(to);
                messageMail.From = new MailAddress(from, displayName);
                messageMail.Subject = subject;
                messageMail.IsBodyHtml = true;
                messageMail.Body = htmlContent;
                if (attachments != null)
                {
                    foreach (var item in attachments)
                    {
                        messageMail.Attachments.Add(new Attachment(item.Value, item.Key));
                    }
                }
                smtpClient.SendCompleted += (s, e) =>
                {
                    smtpClient.Dispose();
                    messageMail.Dispose();
                };

                await smtpClient.SendMailAsync(messageMail);
                smtpClient.Dispose();
                messageMail.Dispose(); 
            }

catch (Exception e)
            {
                MailMessage messageMail = new MailMessage();
                messageMail.To.Add("errors@mydomain");
                messageMail.From = new System.Net.Mail.MailAddress(AppConfig.NoReplyMail);
                messageMail.Subject = "Api Error";
                messageMail.IsBodyHtml = true;
                messageMail.Body = e.Message + "\n " + e.InnerException + "\n" + "to: " + to + "\n subject:" + subject + "\args: " + result;
                using (var smtpClient = new SmtpClient())
                {
                    smtpClient.Send(messageMail);
                }
            }
            }

Yes we understand that we are using "using" and multiple "Dispose", but none of them have any effects. This is just part of our tests.

In this particular case, we are not attaching anything. It is used in other parts of the code where we send only one email out. — WPalombini, May 07 '15 at 06:51
Are you sure you are not falling in infinite loop? Also, are you using optimized size of threads. — Amit, May 07 '15 at 06:59
the service we use is prepared to take hundreds of thousands of emails per second. Also, we get the same error when running from localhost while creating the .eml files in the temp folder, instead of actually emailing them out! — WPalombini, May 07 '15 at 06:59
Yes Amit, I am 100% sure this is not an infinite loop. If we just dont send the email out by commenting the await smtpClient.SendMailAsync(messageMail); line, we have no errrors at all — WPalombini, May 07 '15 at 07:00
Looks quite okay, if you're disposing all (including attachments) disposable objects it should work. Is there more code in the `SendMail` function? Do you dispose database objects as well? — Stefan, May 07 '15 at 07:02
What if you make Task[5000] to Task[100] ? You need to check what is optimized size. — Amit, May 07 '15 at 07:03
We have the error even I make Task[20]. By monitoring my Task Manager, memory increases about 1.3Gb until we have the error. More: If I dont pass the email body in, say if I make the body of the email just a "" string, it works. The email file size is roughly 32kb (the one that should go out to clients) — WPalombini, May 07 '15 at 07:05
Stefan, there is nothing else in the SendMail function, apart from the catch part of the try section. — WPalombini, May 07 '15 at 07:06
`htmlContent = htmlContent.Replace("{" + i + "}", args[i]);`. Assuming htmlContent is a string, how many items are there in args? Huge assumption: too many strings are getting created and either getting allocated on LOH or getting promoted to Gen2. Try using tools like [VMMap](https://technet.microsoft.com/en-us/library/dd535533.aspx) and PerfMon to figure out if this is the case. — publicgk, May 07 '15 at 07:14
that string replace is replacing about 5 params. I have never heard about these tools, will see what I can find. Do you have more details about it? — WPalombini, May 07 '15 at 07:17
@WPalombini: the `catch` statement might be crucial, or is if there's an exception raised. I would suggest to use a `finally` block for the disposure, and post the full code next time ;-) — Stefan, May 07 '15 at 07:18
@publicgk in fact, if I dont do the string replace and always pass the same parameters to all emails (as a test) I still get the errors at the same point. — WPalombini, May 07 '15 at 07:20
By the way, what is: `candidate.FirstOrDefault()`? Is `candidate` a lazy loading object/collection? Why `FirstOrDefault`? Name suggests it should be just one. — Stefan, May 07 '15 at 07:22
I'm voting to close this question as off-topic because it belongs on [CodeReview](http://codereview.stackexchange.com) — Yuval Itzchakov, May 07 '15 at 07:25
@Stefan that is due to a group by in the sql query. Thanks for your help, but that is not the point of the problem. Like I said, if I just pass a small string to the body of the email, it works. The problem is that memory is not being managed properly — WPalombini, May 07 '15 at 07:26
The equation is rather simple. I'm assuming you're using a x86 process, which has to allocate a huge amount of strings (by looking at the concatenation and other string manipulation you're doing). Plus, you spin 5000 threads which all take part in allocating mailmessages and sending them. Your best friend here would be a memory profiler — Yuval Itzchakov, May 07 '15 at 07:27
@YuvalItzchakov we are using azure websites and this is a restful web api. What memory profiler are you refering to? thanks — WPalombini, May 07 '15 at 07:30
There are plenty of good [memory profilers](http://stackoverflow.com/questions/399847/net-memory-profiling-tools) — Yuval Itzchakov, May 07 '15 at 07:36
@WPalombini: Not to be rude but since there is a memory issue here, I will omit assumptions about the validity of your conclusions. Providing full context and code would help to determine the actual problem. For example: the `catch` block is still a crucial part of this piece of code, because, although I see plenty of `Dispose` calls, if it's missing in the `catch` part there will be an issue with these kind of numbers. If I didn't need to spend so much time retrieving this information, I perhaps would have run a profiler myself. Anyways, goodluck. — Stefan, May 07 '15 at 07:42
@WPalombini, if you have never heard of those tools, then this would be a good starting point: https://support.microsoft.com/en-us/kb/2020006. It's says it's for ASP.net, but most of the content is relevant for general .net. In your case, you have ruled out string-concatenation, then the next candidate seems to be fragmentation in managed heap or Virtual Address space. See, if the article helps. If nothing helps, then as suggested by others you might have to resort to Memory Profiler tools (which are generally paid). — publicgk, May 07 '15 at 08:21

Out of Memory sending mass emails using SmtpClient

0 Answers0