0

I am trying to scrape from an MT5 server. On my browser the site shows loading for a few seconds or milliseconds then renders the login form. But with puppeter it doesn't. Just the loading page like forever, till timeout. And when i try to code puppeteer wait until the element is rendered it crashes after timeout. I have tried different methods to find out whats wrong(Some of which i commented out in my code). I don't know where i got it wrong or is it from the site?

This is my first time using Puppeteer too. My Code is below. Any body that can help me out with solutions?

const puppeteer = require("puppeteer");

const baseUrl = "https://trade.mql5.com/trade?servers=";

const getData = async () => {
  console.log(baseUrl);
  // launch pupprteer
  const browser = await puppeteer.launch({ headless: false });
  const page = await browser.newPage();
  //   await page.setDefaultNavigationTimeout(60000);
  await page.goto(baseUrl, { waitUntil: "networkidle2", timeout: 0 });
  //   await page.waitForNavigation(() =>
  //     document.querySelector("#hero-head-circle")
  //   );
  //   await page.screenshot({ path: "login.png" });
  await page.waitForSelector("#login"),
    await page.screenshot({ path: "login.png" });

  await browser.close();
};

getData();
Ebuka
  • 19
  • 1
  • 9
  • Did you try adding a user agent, puppeteer-extra-plugin-stealth or any other [strategies for avoiding detection](https://stackoverflow.com/questions/51731848/how-to-avoid-being-detected-as-bot-on-puppeteer-and-phantomjs)? – ggorlen Jun 20 '22 at 22:03
  • no i didn't. Don't know what that is. This is my first time. I'll look it up – Ebuka Jun 20 '22 at 22:11

1 Answers1

1

I think i had the same issue, somepages load content as you scroll down through them, so the selector #login may not be apppering unless you scroll down; if that is the issue try with this code:

const { scrollPageToBottom } = require('puppeteer-autoscroll-down')       
const lastPosition= await scrollPageToBottom(page, {
            size: 110,
            delay: 250,
        })

This solved the issue for me. Have in mind that you may have to change "size" and "delay" depending on the page that you are using.

Nicolas
  • 11
  • 3