Crawl react pages in Java

Asked Aug 02 '17 at 08:36

Active Aug 02 '17 at 08:36

Viewed 499 times

I'm writing a web crawler in Java. The target is a website written in react. I use selenium to render the page to a DOM tree:

HtmlUnitDriver unitDriver = new HtmlUnitDriver();
unitDriver.get("https://www.instagram.com/9gag/?hl=en");
WebElement body = unitDriver.findElement(By.tagName("body"));

What I get for body is:

<body class="">

Which is almost nothing. Does selenium render react pages ? I also tried this on google.com and it worked fine.

asked Aug 02 '17 at 08:36

Arian

7,397
21
89
177

Is this related to web crawling or selenium? – Nabin Aug 02 '17 at 08:40
I'm using Selenium to do the web crawling – Arian Aug 02 '17 at 08:41
As far as I know Selenium is for surfing; creating bots, agents and automated testing. – Nabin Aug 02 '17 at 08:42
Do you know any other headless browser in Java that renders react pages ? – Arian Aug 02 '17 at 08:44
React revuilds DOM on fly with JS. So initially you have just root element where the rest is added by JS. See https://stackoverflow.com/questions/16073626/how-can-i-get-html-content-written-by-javascript-with-selenium-python – StanislavL Aug 02 '17 at 08:52
@StanislavL how should we tell the headless browser to wait ? is this doable using Selenium ? – Arian Aug 02 '17 at 08:55
yes, selenium has a `pause` function – Krzysztof Cichocki Aug 02 '17 at 08:56
Wait? For what? You can set a timeout e.g. for a second and then use browser object or exec scripts to get DOM elements updated by JS. – StanislavL Aug 02 '17 at 08:58
@StanislavL I tried `Thead.sleep` right after `unitDriver.get` and it didn't help. – Arian Aug 02 '17 at 09:07

Crawl react pages in Java

0 Answers0