1

I am trying to intercept a website - https://www.kroger.com/pl/chicken/05002. In the chrome network tab, I see the request as below, with the details of the products nicely listed as JSON

enter image description here

I copied the cURL as bash and imported it as raw text in Postman. It ran forever without any response. Then I used the intercept feature and still it is running forever.

enter image description here

When both the requests are exactly same, why is it running in Chrome and not in Postman? What am i missing? Any help is appreciated, thanks in advance.

lu5er
  • 3,229
  • 2
  • 29
  • 50

1 Answers1

1

This is probably happening because they don't want you to do what you are trying to do. Note the "filter.verified" param in the URL.

You may want to try reaching out to them for an external API token - especially if you are creating an app or extension to compare competitive prices with the intention of distributing said app or extension - regardless of if it is for financial compensation or not.

Ethically questionable workaround (which would defintely need to be improved upon - this is simply an example of how you could solve your problem...):

GET https://www.kroger.com/search?query=chicken&searchType=default_search&fulfillment=all

const html = cheerio(responseBody);

var results = [];

html.find('div[class="AutoGrid-cell min-w-0"] > div').each(function (i, e)
{
    results.push({
        "Item": e.children[e.children.length-3].children[0].children[0].children[0]["data"],
        "Price": e.children[e.children.length-4].children[0].attribs["value"]
    })
});

console.log(results);

Output

If you are unable to obtain an API token from them, this would probably be a legal way to accomplish what you want.

Stroh
  • 35
  • 5
  • Thanks for your help. Definitely, that can be an option. But I'm not exactly looking for the price but other Product level details which are perfectly coming out in the JSON response such as nutrition, allergy information, etc. It's not exactly that I want to scrape Kroger, but just want to know how they are blocking the calls so that it works in Chrome but not outside and what can be tweaked to make the call. This is just for educational purposes only. – lu5er Apr 09 '21 at 03:48
  • 1
    I more looked into a solution versus the problem itself. I would check cookies if I were you. Additionally, I would be quite surprised if the other attributes of interest weren't as readily parse-able. – Stroh Apr 09 '21 at 08:34
  • Just realized - you could create an HTML template, and use the DOMParser class inside of a script. See: https://stackoverflow.com/questions/63249665/need-to-copy-api-success-response-to-clipboard-in-postman for a template creation example – Stroh Apr 23 '21 at 16:09