0

I am trying to get the company data from this website called similar web but upon making a lot of requests it recognizes my script as a bot so is there any way to bypass this check? or suggest any website to scrap data easily, we can't use LinkedIn by the way.

const puppeteer = require("puppeteer");
const searchCompany = "zoominfo.com";
const Link = `https://www.similarweb.com/website/${searchCompany}/#overview`;
// console.log(companyPage);
let page;
(async function () {
  try {
    let browserOpen = await puppeteer.launch({
      headless: false,
      //   dumpio: true,
      //   args: ["--start-maximized"],
      defaultViewport: null,
    });
    let newTab = await browserOpen.newPage();
    await newTab.goto(Link);
    await newTab.screenshot({ path: "sc.png" });
    await newTab.waitForSelector(".data-company-info__row");
    let ans = await newTab.evaluate(() => {
      let name = document.querySelectorAll(".data-company-info__row")[0]
        .textContent;
      let location = document.querySelectorAll(".data-company-info__row")[3]
        .textContent;
      let industry = document.querySelectorAll(".data-company-info__row")[5]
        .textContent;
      //   console.log(ans);
      return { name, location, industry };
    });
    console.log(ans);
    await browserOpen.close();
  } catch (err) {
    console.log(err);
  }
})();
Afsanefda
  • 3,069
  • 6
  • 36
  • 76
Sam
  • 1
  • 3
  • Your IP might have blocked. Try from a different IP. Dont spam the webserver. Your hits should have some delay between consecutive requests. – kiranvj May 24 '22 at 05:30
  • how do i achieve this ? any ideas @kiranvj – Sam May 24 '22 at 05:32

1 Answers1

0

Just out of curiosity - what do you use similarweb data for?

You can try using https://github.com/bda-research/node-crawler that has delays and max connections params

tamirp
  • 31
  • 2
  • Your answer could be improved with additional supporting information. Please [edit] to add further details, such as citations or documentation, so that others can confirm that your answer is correct. You can find more information on how to write good answers [in the help center](/help/how-to-answer). – Community May 24 '22 at 07:32