How to send post data with WebKitFormBoundary params in puppeteer?

Question

I try to make a post request to some API that requires WebKitFormBoundary params as form data and just can't find out how to get it done. I'm also not sure how the random(?) string following WebKitFormBoundary is made up and it could be an issue, but I'm not sure.

So I copied the request from dev tools. Here is the relevant part of my code for the request headers:

(async () => {
    const browser = await puppeteer.launch();
    let page = await browser.newPage();
    await page.setRequestInterception(true);
    let header = {
        'method': 'POST',
        'path': '/market/content/marketsearch',
        'content-type': 'multipart/form-data; boundary=----WebKitFormBoundaryFWRUYTIwAHUwxud6',
    };
    await page.setExtraHTTPHeaders(header);

As you can see, the content-type perfectly shows what the WebKitFormBoundary has to consist of. In this case "FWRUYTIwAHUwxud6". I'd really appreciate if someone could explain the logic, if there is any, behind that string.

Next thing I do is to interpect the request and input the form data. Here is my first attempt that obviously failed.. Server can't handle new line (\n) and returns a http status code of 500. This is the exact source from payload tab in dev tools.

await page.on('request', interceptedRequest => {
    let data = {
        'method': 'POST',
        'postData': '------WebKitFormBoundaryFWRUYTIwAHUwxud6\n' +
                    'Content-Disposition: form-data; name="searchString"\n' +
                    '\n' +
                    'ABC\n' +
                    '------WebKitFormBoundaryFWRUYTIwAHUwxud6\n' +
                    'Content-Disposition: form-data; name="page"\n' +
                    '\n' +
                    '0\n' +
                    '------WebKitFormBoundaryFWRUYTIwAHUwxud6\n' +
                    'Content-Disposition: form-data; name="city"\n' +
                    '\n' +
                    '\n' +
                    '------WebKitFormBoundaryFWRUYTIwAHUwxud6\n' +
                    'Content-Disposition: form-data; name="pageSize"\n' +
                    '\n' +
                    '50\n' +
                    '------WebKitFormBoundaryFWRUYTIwAHUwxud6--'
    }
    interceptedRequest.continue(data);
});

So i tried the parsed and decoded version next but wasn't sure how exactly to pass the data. I tried 2 versions but neither worked.

    let data = {
        'method': 'POST',
        'postData': 'searchString=ABC&page=0&city=&pageSize=50'
    }

This returned an empty result of :

<html><head><meta name="color-scheme" content="light dark"></head><body><pre style="word-wrap: break-word; white-space: pre-wrap;">{"markets":[],"total":0,"nextPage":null,"prevPage":null}</pre></body></html>

And so did the second attempt.

    let data = {
        'method': 'POST',
        'postData': {
              searchString: 'ABC',
              page: 0,
              city: '',
              pageSize: 50
        }
    }

I am thankful for every piece of help and advice I and can get. For the sake of clarity, I'll include the whole code in one piece.


(async () => {
    const browser = await puppeteer.launch();

    let page = await browser.newPage();

    await page.setRequestInterception(true);

    let header = {
        'method': 'POST',
        'path': '/market/content/marketsearch',
        'scheme': 'https',
        'accept': '*/*',
        'accept-encoding': 'gzip, deflate, br',
        'accept-language': 'de,en-US;q=0.9,en;q=0.8,de-DE;q=0.7',
        'content-type': 'multipart/form-data; boundary=----WebKitFormBoundaryFWRUYTIwAHUwxud6',
        'sec-ch-ua': '"Chromium";v="106", "Google Chrome";v="106", "Not;A=Brand";v="99"',
        'sec-ch-ua-mobile': '?0',
        'sec-ch-ua-platform': '"Windows"',
        'sec-fetch-dest': 'empty',
        'sec-fetch-mode': 'cors',
        'sec-fetch-site': 'cross-site',
        'user-agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/106.0.0.0 Safari/537.36'
    };

    await page.setExtraHTTPHeaders(header);

    await page.on('request', interceptedRequest => {
        let data = {
            'method': 'POST',
            'postData': '?'
        }
        interceptedRequest.continue(data);
    });

    await page.goto(TARGET_URL, {waitUntil: 'domcontentloaded'});

    console.log(await page.content());

    await browser.close();
})();

Thanks a lot

score 0 · Answer 1 · answered Nov 04 '22 at 15:28

Okey, I solved the issue and will answer it here, in case anyone has the same issue in the future.

First question:

As you can see, the content-type perfectly shows what the WebKitFormBoundary has to consist of. In this case "FWRUYTIwAHUwxud6". I'd really appreciate if someone could explain the logic, if there is any, behind that string.

Answer: This post sums it up really nice.

You specify it in the Content-Type header so that the server knows how to split the data it receives. https://stackoverflow.com/a/20321259/13940724.

Solution to Issue: >This< peace of information helped me lot. In short, "As with all MIME transmissions, "CR LF" (i.e., `%0D%0A') is used to separate lines of data." got me thinking. I added "\r" to all my "\n" in postData. Here's the code:

await page.on('request', interceptedRequest => {
    interceptedRequest.continue(
        {
            method: 'POST',
            postData: '------WebKitFormBoundaryl0BgvXXgOBvseuQF\r\n' +
                'Content-Disposition: form-data; name="searchString"\r\n' +
                '\r\n' +
                'ABC\r\n' +
                '------WebKitFormBoundaryl0BgvXXgOBvseuQF\r\n' +
                'Content-Disposition: form-data; name="page"\r\n' +
                '\r\n' +
                '0\r\n' +
                '------WebKitFormBoundaryl0BgvXXgOBvseuQF\r\n' +
                'Content-Disposition: form-data; name="city"\r\n' +
                '\r\n' +
                '\r\n' +
                '------WebKitFormBoundaryl0BgvXXgOBvseuQF\r\n' +
                'Content-Disposition: form-data; name="pageSize"\r\n' +
                '\r\n' +
                '1\r\n' +
                '------WebKitFormBoundaryl0BgvXXgOBvseuQF--',
        });
});

Worked like a charm.

How to send post data with WebKitFormBoundary params in puppeteer?

1 Answers1