Folding JSON strings to send in header variables - Mailgun

Question

I'm trying to use the JS Mailgun API to send emails. Have it working fine, until I throw template variables into 'h:X-Mailgun-Variables' like so, where jsonString is very large (17000+ characters):

const mailData = {
    from: 'Insights <insights@hello.net>',
    to: mailAddress,
    subject: `Insights: ${DAYS_OF_WEEK[date.getDay()]}, ${MONTHS[date.getMonth()]} ${ordinal_suffix_of(date.getDate())}  ${date.getFullYear()}`,
    template: "template1",
    'h:X-Mailgun-Variables': jsonString,
  };

Looking at the documentation here states the following:

Note The value of the “X-Mailgun-Variables” header must be valid JSON string, 
otherwise Mailgun won’t be able to parse it. If your X-Mailgun-Variables 
header exceeds 998 characters, you should use folding to spread the variables 
over multiple lines.

Referenced this post, which suggested I "fold" up the JSON by inserting CRLF characters at regular intervals. This led me here, which still does not work, though logging this does show regular line breaks and is compliant JSON:

const jsonString = JSON.stringify(templateVars).split('},').join('},\r \n');

Any insight into how to properly "fold" my JSON so I can use large template variables in my MailGun emails?

UPDATE:

As requested, adding my code. This works when data only has a few companies/posts, but when I have many companies each with many posts, I get the 400 error:

function dispatchEmails(data) {
  const DOMAIN = 'test.net';
  const mg = mailgun({apiKey: API_KEY, domain: DOMAIN});
  
  const templateVars = {
    date: theDate,
    previewText: 'preview',
    subject: 'subject',
    subhead: 'subhead',
    companies: data.companies.map(company => {
      return {
        url: company.url,
        totalParts: data.totalParts,
        currentPart: data.currentPart,
        companyData: {
          name: company.name,
          website: company.website,
          description: company.description
        },
        posts: _.map(company.news, item => {
          return {
            category: item.category,
            date: new Date(item.date),
            url: item.sourceUrl,
            title: item.title,
            source: item.publisherName,
            description: item.description,
          }
        })
      }
    })
  };

  const jsonString = JSON.stringify(templateVars).split('},').join('},\r \n');

  const mailData = {
    from: 'test@test.com',
    to: 'recipient@test.com',
    subject: 'subject',
    template: 'template',
    'h:X-Mailgun-Variables': jsonString
  };

  return mg.messages().send(mailData)
  .then(body => {
    return body;
  })
  .catch(err => {
    return {error: err};
  });
}

Have you taken a look at this? https://stackoverflow.com/questions/9779860/using-json-string-in-the-http-header/40415268 — TechySharnav, Mar 07 '21 at 16:36
I have - tried those encoding methods, but still no dice. I know my json is formatted correctly because it goes through if I throw in a shorter version. But when the object is large, it errors out. So looking for a way to properly "fold" my json to conform to Mailgun's spec. I feel like I've implemented correctly now (see post), but still isn't working. — snn, Mar 07 '21 at 16:52
This may not always work because folding will have a requirement that your object's data is not large. You need to provide a sample json structure at least in your question — Tarun Lalwani, Mar 08 '21 at 13:09
@TarunLalwani just updated with my code that generates JSON. I tried folding because the Mailgun documentation told me to use it if I was sending over long JSON, so assumed that would be the solution. But open to advice/suggestions! — snn, Mar 08 '21 at 19:31
How about using the approach list in this ? https://stackoverflow.com/questions/50155202/json-stringify-how-to-skip-indentation-for-one-or-more-objects for the `"posts:"` — Tarun Lalwani, Mar 09 '21 at 05:03
@TarunLalwani Yup! Tried removing whitespace as well. Unfortunately did not help — snn, Mar 09 '21 at 23:52
Can you try `const jsonString = JSON.stringify(templateVars, null, 1);`? — Tarun Lalwani, Mar 10 '21 at 14:56
@TarunLalwani Unfortunately same result. Works with shorter objects, errors out on longer. — snn, Mar 10 '21 at 20:37
Should you have a space between `\r` and `\n` ? Or you meant to write `JSON.stringify(templateVars).split('},').join('},\r\n');` ? — Bob, Mar 11 '21 at 10:15
@TarunLalwani trying! Mailgun support has been unresponsive. Appreciate your help though regardless. — snn, Mar 11 '21 at 19:51
@user12750353 I've tried both "/r/n" and "/r /n" - I tried it with the space because of the other stackoverflow question I linked, which stated: "Section 2.2.3 talks about long header fields, > 998 characters, and says such headers need to be folded by inserting the CRLF characters followed immediately by some white space, eg a space character." — snn, Mar 11 '21 at 19:52
CRLF followed immediately by a space would be `\r\n ` (space after `\n`), I don't know this folding technique, just guessing here. I would send the JSON in the request body. — Bob, Mar 11 '21 at 19:59

score 1 · Answer 1 · answered May 03 '22 at 14:58

I think your problem may be overall payload size rather than the string folding. The folding for strings exceeding 998 characters seems to be handled by the node.js client, possibly by the form-data package.

I ran the following test:

test_big_data (template created in mailgun through the UI)

<!DOCTYPE html>
<html>
    <body>
        <h2>An HTML template for testing large data sets.</h2>
        <ul>
            <li>{{param_name_0}}</li>
                ... other lis ...
            <li>{{param_name_999}}</li>
        </ul>
    </body>
</html>

send_email.js

const API_KEY = 'MY_KEY';
const DOMAIN = 'MY_DOMAIN.COM';

const formData = require('form-data');
const Mailgun = require('mailgun.js');

const mailgun = new Mailgun(formData);
const client = mailgun.client({ username: 'api', key: API_KEY });

const bigData = {};

for (let i = 0; i < 400; i++) {
    bigData[`param_name_${i}`] = `param_value_${i}`;
}

const dataString = JSON.stringify(bigData);
console.log(dataString.length);

const messageData = {
    from: 'Mailgun Sandbox <postmaster@DOMAIN>',
    to: 'test@MY_DOMAIN.COM',
    subject: 'Big Data Test',
    template: 'test_big_data',
    'h:X-Mailgun-Variables': dataString,
};

client.messages
    .create(DOMAIN, messageData)
    .then((res) => {
        console.log(res);
    })
    .catch((err) => {
        console.error(err);
    });

The length of the dataString in this case is 13781 characters and the email queues successfully.

However, if I bump the for loop condition to i < 1000 I get the following error when queueing the email:

[Error: Bad Request] {
  status: 400,
  details: '{"message":"Send options (parameters starting with o:, h:, or v:) are limited to 16 kB total"}\n'
}

When I asked Mailgun support about the folding warning form the documentation they pointed me to RFC 2822 section "3.2.3. Folding white space and comments". But like I said, I don't think folding is the issue here.

Cheers! https://datatracker.ietf.org/doc/html/rfc2822#page-11

score 0 · Answer 2 · answered Mar 14 '21 at 14:03

Just thinking outside the box but why pass that much data in an email header? I assume you have something on the receiving end which is going to parse the email headers. What if instead of sending them that data you send them a key that they can call back into an API on your end to get the data

Folding JSON strings to send in header variables - Mailgun

2 Answers2