Kotlin, how can I read a dynamic website as text?

Question

As titled, I'm trying to read the content of sites like this one, which appears to be javascript based.

I tried using plain jdk lib, then jsoup and then htmlunit, but I couldn't get anything useful out of it (I see just the source code or just the title or null):

val url = URL("https://registry.terraform.io/providers/hashicorp/tls/latest/docs/data-sources/certificate")
val connection = url.openConnection()
val scanner = Scanner(connection.getInputStream())
scanner.useDelimiter("\\Z")
val content = scanner.next()
scanner.close()
println(content)

val doc = Jsoup.connect("https://registry.terraform.io/providers/hashicorp/tls/latest/docs/data-sources/certificate").get()
    println(doc.text())

WebClient().use { webClient ->
    val page = webClient.getPage<HtmlPage>("https://registry.terraform.io/providers/hashicorp/tls/latest/docs/data-sources/certificate")
    val pageAsText = page.asNormalizedText()
    println(pageAsText)
}

WebClient(BrowserVersion.FIREFOX).use { webClient ->
    val page = webClient.getPage<HtmlPage>("https://registry.terraform.io/providers/hashicorp/tls/latest/docs/data-sources/certificate")
    println(page.textContent)
}

It should be something easy peasy, but I cant see what's wrong

You will need to use a framework that also does the JavaScript execution, I remember that Selenium could do it. — Horațiu Udrea, Jul 04 '22 at 17:30

Joffrey · Answer 1 · 2022-07-04T21:23:31.150

1

In order for this to be possible, you need something to execute the JS that modifies the DOM.

It might be a bit overkill depending on the use case, and probably won't be possible if you're on Android, but one way to do this is to launch a headless browser separately and interact with it from your code. For instance, using Chrome Headless and the Chrome DevTools Protocol. If you're interested, I have written a Kotlin library called chrome-devtools-kotlin to interact with a Chrome browser in a type-safe way.

There might be simpler options, though. For instance maybe you can run an embedded browser instead with JBrowserDriver and still use JSoup to parse the HTML, as mentioned in this other answer.

edited Jul 04 '22 at 21:23

answered Jul 04 '22 at 17:33

Joffrey

32,348
6
68
100

It sounds interesting, would you mind showing me a code snippet? – elect Jul 04 '22 at 17:54
There are some (admittedly very light) usage examples in the README, for instance for accessing the DOM: https://github.com/joffrey-bion/chrome-devtools-kotlin#connecting-to-targets. – Joffrey Jul 04 '22 at 21:25
I edited my answer with a pointer to another answer suggesting the Java library `JBrowserDriver`, which should provide the same kind of capability but with an embedded browser instead of having to spawn a separate process for the headless Chrome. The API will likely be slightly higher level, but Java-like. I have never used it, though. – Joffrey Jul 04 '22 at 21:26
Thanks, it looks like this is gonna be more difficult than I originally thought: issue [here](https://github.com/MachinePublishers/jBrowserDriver/issues/344) and [here](https://github.com/MachinePublishers/jBrowserDriver/issues/341) – elect Jul 05 '22 at 07:51
Looking from another point of view, I wonder if there is a website that given the address target, could resolve javascript and return the plain text easy to read/parse.. – elect Jul 05 '22 at 07:52

score 0 · Answer 2 · answered Jul 05 '22 at 05:21

Regarding HtmlUnit:

the page has initially no content, all you see is rendered from javascript magic on the client side using one of this spa frameworks. It looks like there is some feature check in the beginning that figures out the js support in HtmlUnit does not have all the required features and based on this you only get a hint like "Please enable Javascript to use this application".

You can use

page.asXml()

to have a look at the content trough HtmlUnit's eyes.

You can open an HtmlUnit issue on github but i fear adding support for this will be a longer story.

Kotlin, how can I read a dynamic website as text?

2 Answers2