Unable to scrape despite IP rotation

Question

I need to scrape this page (the ads): https://www.sahibinden.com/en/cars/used?date=1day&a5_min=2005&a5_max=2020

When I open it too many times I get blocked, changing the IP doesn't help either. The problem is that when I open this page from browser from my PC it works just fine. But it seems to get blocked from webkit.

await page.route("**/*", (route) => {
    if (!firstReq) route.continue();
    else {
      firstReq = false;
      route.continue({
        method: method,
        postData: data,
        headers: headers,
      });
    }
  });
  let pageRes = await page.goto(url);
  await page.waitForNavigation()
  await page.unroute("**/*");
  return pageRes;

I realize that it is site trying to block bots but what are the practices to avoid that. I have tried waits, ip rotation as well as useragent rotation - nothing seems to be working

Does this answer your question? [How to deal with the captcha when doing Web Scraping in Puppeteer?](https://stackoverflow.com/questions/55493536/how-to-deal-with-the-captcha-when-doing-web-scraping-in-puppeteer) — tripleee, Jul 07 '20 at 10:06

score 1 · Accepted Answer · answered Jun 27 '20 at 16:40

In their Terms of Use §4.11 they state that it is not allowed to scrape their content:

The use of the whole or any part of the "Portal" for [...] Automatic program on the site, robot, spider, web crawler , spider, data mining, data crawling etc. "screen scraping" software or systems, using automated tools or manual processes, [...] such uses will be prevented at the discretion of the OWNER. [...]

So you can be sure they are doing their best to prevent scraping.

There are methods to workaround the blocks, I advise you to read Thomas Dondorf's great answer on the topic of headless browsers and blocking by reCaptcha. I also strongly recommend to consider his first option in the current case:

Option 1: Stop crawling or try to use an official API. As the owner of the page does not want you to crawl that page, you could simply respect that decision and stop crawling. Maybe there is a documented API that you can use.

In general there can be huge differences in scraper recognition between visiting the site in headless vs headful mode, using slowMo option of launch() or not.

Unable to scrape despite IP rotation

1 Answers1