Calling jQuery Ajax in a loop is giving me 503 errors

Question

I'm using jQuery Ajax to generate reports and email those reports to customers.

PHP 5.6.31
MySQL 5.5.45
Windows Server 2016
jQuery 1.11.1

If I have a small amount of customers (like 50) it's no problem, but if I have 100 or more then I start getting 503 errors. The first bunch will email no problem, but at a certain point all the "pending" calls to the email script report 503 errors. Usually around the 45 second mark.

Here's my javascript

function emailInvoices(){

  $('#emailreportbutton').button("option","disabled", true);
  $('#email-progresslabel').text( "Starting..." );

  var recipientarray = $('#recipientarray').val();
  var recipients = JSON.parse(recipientarray);
  var recipientcount = recipients.length;

  var totaldueoption = $('#totaldueoption').val();
  var showaccountnumber = $('#showaccountnumber').val();
  var showmessage = $('#showmessage').val();
  var message = $('#message').val();
  var currentcount = 1;
  var successcounter = 0;
  var failcounter = 0;

  //<!--*** For each invoice returned... ***-->

  for (var i=0; i<recipientcount; i++){
    var obj = recipients[i];

    //<!--*** Generate and email the pdf ***-->

    $.ajax({
          type: 'POST',
          url: 'send-invoice.php',
          data: {
          familyid:recipients[i].familyid,
          invoicenumber:recipients[i].invoicenumber,
          motheremail:recipients[i].motheremail,
          fatheremail:recipients[i].fatheremail,
          primarypayer:recipients[i].primarypayer,
          primarypayeremail:recipients[i].primarypayeremail,
          invoicedate:recipients[i].invoicedate,
          totaldueoption:totaldueoption,
          showaccountnumber:showaccountnumber,
          showmessage:showmessage,
          message:message
          },
          success: function(data) {
            if(data == "success"){
              successcounter ++;
              $('#successcount').html(successcounter + " Sent");
              if(failcounter == 0) $('#successcount').addClass('emailsuccess');
            }
            if(data == "fail"){
              failcounter ++
              $('#failcount').html(failcounter + " Failed");
              $('#failcount').addClass('emailfail');
              $('successcount').removeClass('emailsuccess');
            }

            //<!--*** Update progress bar ***-->

            var percentComplete = Math.round(currentcount * 100 / recipientcount);
            $('#progressbar').progressbar("value", percentComplete);

            currentcount ++;

          },
          error: function() {
          // Failed to call send-invoice.php
          }
    });

  }

  $('#reportlink').html("<a href=# onclick=runReport('emailhistory')>Click here to view your email history report</a>");
}

It always worked before but I have been changing some server settings over the past few months to try to resolve an unrelated issue so I'm wondering if I changed any PHP setting to introduce this problem.

I'm not sure how calling ajax in a loop like this is impacted by server settings so I've been stabbing in the dark a bit. In an effort to fix this problem I did the following:

I increased memory_limit to 512MB from 128MB
I increased max_input_time to 60 from 20
I increased post_max_size to 60M from 20M
I increased upload_max_filesize to 30M from 10M
I increased max_file_uploads to 60 from 20

All to no avail.

Here's a screenshot of my chrome developer tools so you can see exactly what I mean. Don't worry about the red warning about failed emails - those are just ones that have no email address. The real problem is those calls to send-invoice.php with a 503 error. The interesting thing is that this error shows up on those files before they're even processed. They all go from pending to error at approximately the same time as if at a certain point the server just says "I'm done - no more".

Not sure if the contents of send-invoice.php is relevant but here it is anyway:

<?php

include("common/common.php");

$familyid = $_POST["familyid"];
$invoicenumber = $_POST["invoicenumber"];
$motheremail = trim($_POST["motheremail"]);
$fatheremail = trim($_POST["fatheremail"]);
$primarypayer = trim($_POST["primarypayer"]);
$primarypayeremail = trim($_POST["primarypayeremail"]);
$attachmentdate = $_POST["invoicedate"];

$totaldueoption = $_POST["totaldueoption"];
$showaccountnumber = $_POST["showaccountnumber"];
$showmessage = $_POST["showmessage"];
$message = $_POST["message"];
$dosend = false;

//<!--********************************************************************************************************-->
//<!-- Get family name -->
//<!--********************************************************************************************************-->

$sql = "select name from families where id = ".$familyid;
$result = mysql_query($sql);

if ($row = mysql_fetch_array($result)){
  $familyname = $row["name"];
}

//<!--********************************************************************************************************-->
//<!-- Get email body -->
//<!--********************************************************************************************************-->

$sql = "select emailbodyinvoice from preferences where companyid = ".$companyid;
$result = mysql_query($sql);

if ($row = mysql_fetch_array($result)){
  $emailbody = $row["emailbodyinvoice"];
}

//<!--********************************************************************************************************-->
//<!-- Generate pdf -->
//<!--********************************************************************************************************-->

include("common/pdf/mpdf/mpdf.php");
ob_start();
$report = "invoice";
$selectedinvoice = $invoicenumber;
include("report.php");
$reporthtml = ob_get_clean();

$style = "
<style>
@page {
  margin: 0px;
}
</style>";

$html = $style.$reporthtml;

$mpdf=new mPDF('c');

$mpdf->mirrorMargins = true;
$mpdf->SetDisplayMode('fullpage','two');
$mpdf->WriteHTML($html);

$invoice = $mpdf->Output('Invoice '.$attachmentdate,'S');

//<!--********************************************************************************************************-->
//<!-- Send invoice email -->
//<!--********************************************************************************************************-->

require('common/html2text.php');

$emailbody = rawurldecode(html_entity_decode("<html><body>".$emailbody."<br><br><div style='color:#929292'>This email was sent on behalf of ".$companyname.".  You may reply to this message to contact ".$companyname." but do not use the sender address (no-monitor@timesavr.net) as that mailbox is not monitored and ".$companyname." will not receive your message.</div></body></html>"));

$emailtextbody = html2text(html_entity_decode($emailbody));
$emailsubject = "Invoice from ".$companyname;

//<!--********************************************************************************************************-->
//<!-- Include dependencies
//<!--********************************************************************************************************-->

require("common/smtpemail.php");

//<!--********************************************************************************************************-->
//<!-- Email sender details
//<!--********************************************************************************************************-->

$mail->From     = "no-monitor@timesavr.net";  // Approved sending domain for SendGrid so the emails don't get flagged as spam
$mail->FromName = $companyname;
$mail->AddReplyTo($companyemail,$companyname);
$mail->AddStringAttachment($invoice,'Invoice '.$attachmentdate.'.pdf');

//<!--********************************************************************************************************-->
//<!-- Add recipients
//<!--********************************************************************************************************-->

$mothervalid = validateEmail($motheremail);
$fathervalid = validateEmail($fatheremail);
$primarypayervalid = validateEmail($primarypayeremail);

if($emailinvoicesto == "P"){
  if($primarypayervalid){
    $mail->AddAddress($primarypayeremail,$primarypayeremail);
    $recipient = $primarypayeremail;
    $dosend = true;
  }
}

if($emailinvoicesto == "M" or $emailinvoicesto == "B"){
  if($mothervalid){
    $mail->AddAddress($motheremail,$motheremail);
    $recipient = $motheremail;
    $dosend = true;
  }
}

if($emailinvoicesto == "F" or $emailinvoicesto == "B"){
  if($fathervalid){
    $mail->AddAddress($fatheremail,$fatheremail);
    if($recipient <> ""){
      $recipient .= ";".$fatheremail;
    }else{
      $recipient .= $fatheremail;
    }
    $dosend = true;
  }
}

//<!--********************************************************************************************************-->
//<!-- Send email
//<!--********************************************************************************************************-->

$emailsubject = htmlentities($emailsubject,ENT_QUOTES);
$familyname = htmlentities($familyname,ENT_QUOTES);

if($dosend){
  if($mail->Send()){
    recordEmail("I",$emailsubject,$familyname,$recipient,"S");
    $result = "success";

  }else{
    recordEmail("I",$emailsubject,$familyname,$recipient,"F");
    $result = "fail";

  }
}else{
  recordEmail("I",$emailsubject,$familyname,$recipient,"F","No email address found");
  $result = "fail";
}

echo $result;

mysql_close();
?>

You simply need to reduce the number of calls being made at a time. You can do this by either batching them (requires changes to php logic) or by throttling the requests you are sending (avoid more than n pending requests at once.) — Kevin B, Sep 28 '18 at 20:06
I don't know if it is something php would support, but my experience with java servers/servlets would suggest that the best way to do this is to parse all of those emails that you need to send into an object or array and send all of that in one `$.ajax` call. Then, your server can iterate through the addresses and send them with out being restricted by the client side. — Rich, Sep 28 '18 at 20:17
Rich, perhaps I could send all the data to the php script and do the loop in there instead of in the client. The one drawback I can see there is I would lose the ability to update a progress bar. — Vincent, Sep 28 '18 at 20:23
Kevin, I'm considering your suggestion to send the invoices in bunches but let's say I pass in 50 invoices to my function, how will I know when all the ajax calls for those 50 are complete before I send another 50? — Vincent, Sep 28 '18 at 20:24
Perhaps take a look at [this SO link](https://stackoverflow.com/questions/7049303/show-progress-for-long-running-php-script)? It seems with some changes you could keep track of the progress with a separate process. Unfortunately, I can't give you any specific php related advice as I haven't worked with php since an introductory course in college. — Rich, Sep 28 '18 at 20:31
I think if it was me, I'd have the server iterate through the mails and keep track of the progress through some global variable. Using a separate URL, you could intermittently call that URL for updates on the progress variable. — Rich, Sep 28 '18 at 20:34

score 1 · Accepted Answer · answered Sep 29 '18 at 16:23

What I ended up doing was to get rid of the loop and only send the next email when the previous one was done. It's basically a callback on success. So this way I don't have a bunch of ajax calls executing at the same time, but one at a time.

It achieves the same thing as synchronous execution (async = false), but this way it doesn't lock up the UI and I can still increment the progress bar.

//<!--********************************************************************************************************-->
//<!-- emailInvoices()
//<!--********************************************************************************************************-->

function emailInvoices(){

  var recipientarray = $('#recipientarray').val();
  var recipients = JSON.parse(recipientarray);
  var recipientcount = recipients.length;

  //<!--*** Pass in recipient array, first index and total recipient count... ***-->
  emailInvoice(recipients,0,recipientcount);
}

//<!--********************************************************************************************************-->
//<!-- emailInvoice(recipient,i,recipientcount)
//<!--********************************************************************************************************-->


function emailInvoice(recipients,i,recipientcount){

  $.ajax({
        type: 'POST',
        url: 'send-invoice.php',
        data: {
          familyid:recipients[i].familyid,
          invoicenumber:recipients[i].invoicenumber,
          motheremail:recipients[i].motheremail,
          fatheremail:recipients[i].fatheremail,
          primarypayer:recipients[i].primarypayer,
          primarypayeremail:recipients[i].primarypayeremail,
          invoicedate:recipients[i].invoicedate,
          totaldueoption:totaldueoption,
          showaccountnumber:showaccountnumber,
          showmessage:showmessage,
          message:message
        },
        success: function(data) {

          //<!--*** Update progress bar ***-->
          var percentComplete = Math.round((i+1) * 100 / recipientcount);
          $('#progressbar').progressbar("value", percentComplete);

          //<!--*** Increment index and call this function again -->
          i++;
          if(i < recipientcount){
            emailInvoice(recipients,i,recipientcount);
          }
        }
   });
}

The key to making this work is to increment the index that I pass in each time I call the function.

Calling jQuery Ajax in a loop is giving me 503 errors

1 Answers1